Skip to content

pytest arg parsing bug #21176

Open
Open
pytest arg parsing bug#21176
Assignees
eleanorjboyd
Labels
area-testingbugIssue identified by VS Code Team member as probable bugneeds spikeLabel for issues that need investigation before they can be worked on.

Description

@philippun1
Philipp (philippun1)
opened on May 3, 2023

Type: Bug

Behaviour

Expected vs. Actual

Running a pytest test with pytestArgs results in the error

Error while running tests: test_example
TypeError: Cannot read properties of null (reading 'testsuites')

After removing the pytestArgs from the settings.json, everything runs as expected.

Steps to reproduce:

  1. Open a project with a pytest (i.e. https://github.com/philippun1/pytest-vscode-parameters which I created specifically to reproduce this bug)
  2. Let VS Code create a settings.json (or use the one from my repo)
  3. Add pytestArgs (or use the one from my repo), i.e.
"python.testing.pytestArgs": [
    ".",
    "--test_option",
    "somevalue"
  ],
  1. Go to the testing tab and run the test
  2. Test Results tab shows the type error
  3. Run from the terminal instead gives no error: pytest . --test_option somevalue

Diagnostic data

This test report was made on an Ubuntu machine, but can also be reproduced on a Windows system with the latest VS Code and python extension.

  • Python version (& distribution if applicable, e.g. Anaconda): 3.11.2
  • Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Global
  • Value of the python.languageServer setting: Default
Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python) 

[output is empty]

User Settings 


languageServer: "Pylance"

testing
• pytestArgs: "<placeholder>"
• pytestEnabled: true

Extension version: 2023.6.1
VS Code version: Code 1.77.3 (704ed70d4fd1c6bd6342c436f1ede30d1cff4710, 2023-04-12T09:16:52.732Z)
OS version: Linux x64 6.2.0-20-generic
Modes:
Sandboxed: Yes

System Info
Item Value
CPUs Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz (8 x 2804)
GPU Status 2d_canvas: enabled
canvas_oop_rasterization: disabled_off
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_renderer: enabled_on
video_decode: disabled_software
video_encode: disabled_software
vulkan: disabled_off
webgl: enabled
webgl2: enabled
webgpu: disabled_off
Load (avg) 1, 1, 1
Memory (System) 15.35GB (8.29GB free)
Process Argv --unity-launch --crash-reporter-id 2d6dab3d-7a47-4211-935d-970af4709a92
Screen Reader no
VM 0%
DESKTOP_SESSION ubuntu
XDG_CURRENT_DESKTOP Unity
XDG_SESSION_DESKTOP ubuntu
XDG_SESSION_TYPE wayland
A/B Experiments 
vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263cf:30335440
vscorecescf:30445987
pythondataviewer:30285071
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30581797
vsaa593:30376534
pythonvs932:30410667
cppdebug:30492333
vsclangdc:30486549
c4g48928:30535728
dsvsc012:30540252
pynewext54:30695312
azure-dev_surveyone:30548225
3biah626:30602489
pyind779:30671433
f6dab269:30613381
vscrpc:30673769
pythonsymbol12:30671437
6233i204:30730053
vsctsb:30705553
azdwalk:30721579
pythonms35:30701012
pythonfmttextcf:30716743
pythonclientmvcf:30725893
fixshowwlkth:30730052

Activity

  1. github-actions
    added
    triage-neededNeeds assignment to the proper sub-team
    on May 3, 2023
  2. Karthik Nadig (karthiknadig)
    assigned
    Eleanor Boyd (eleanorjboyd)
    on May 4, 2023

  3. eleanorjboyd commented on May 4, 2023

    @eleanorjboyd
    Eleanor Boyd (eleanorjboyd)
    on May 4, 2023
    Member

    Hi! Yes I am able to replicate what you are seeing. It seems like the extension is trying to run ./.venv/bin/python -m pytest --rootdir . --override-ini junit_family=xunit1 --junit-xml=/var/folders/q8/17wzg0cx7p59c1qfvwxx8yhc0000gn/T/tmp-852821jmb8LyMYWVJ.xml --test_option which does not correctly include the "somevalue" parameter. This is likely due to the parsing we have on our current test design which is buggy. We are in the process of doing a testing rewrite which you can follow along with here. It is almost ready for release and I have already tested it with your use case and it works successfully. Please subscribe to the issue for updates and we will hopefully have more information soon!

  4. github-actions
    added
    info-neededIssue requires more information from poster
    on May 4, 2023
  5. Eleanor Boyd (eleanorjboyd)
    added
    bugIssue identified by VS Code Team member as probable bug
    area-testing
    and removed
    info-neededIssue requires more information from poster
    triage-neededNeeds assignment to the proper sub-team
    on May 4, 2023

  6. eleanorjboyd commented on Jul 7, 2023

    @eleanorjboyd
    Eleanor Boyd (eleanorjboyd)
    on Jul 7, 2023
    Member

    Hello! We have just finished our testing rewrite and are beginning the roll out to users. If you are able, it would be very helpful to know if your issue still exists on the rewrite! To try it yourself, you need to be on the most recent release of the Python extension and add this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"]. Let me know if the rewrite fixes your issue. Thanks!

  7. philippun1 commented on Jul 7, 2023

    @philippun1
    Philipp (philippun1)
    on Jul 7, 2023
    Author

    It appears there is no difference if I add the pythonTestAdapter setting or not. Tested both with the latest Python Plugin release (v2023.12.0) and preview release (v2023.13.11881007) with VS Code 1.80.0

    Can I somehow verify if the new test adapter is being used?

    I updated my test repo with the pythonTestAdapter setting and also some updates to the test itself to make analyzing easier.

  8. eleanorjboyd commented on Jul 7, 2023

    @eleanorjboyd
    Eleanor Boyd (eleanorjboyd)
    on Jul 7, 2023
    Member

    Hi! You can see if it is enabled for you by setting "python.analysis.logLevel": "Trace", in your user settings then check for 2023-07-05 16:06:27.767 [info] Experiment 'pythonTestAdapter' is active in your python logs. If that message is printed then the rewrite is enabled. I will try and repro this later but if you send over your python logs then I might be able to diagnose it quicker. Thanks for your help!!

  9. philippun1 commented on Jul 7, 2023

    @philippun1
    Philipp (philippun1)
    on Jul 7, 2023
    Author

    This is my python log if I run the test in the Testing tab:

    2023-07-07 22:38:26.077 [warning] Unknown command line option passed into args parser for tests '--test_option'. Please report on https://github.com/Microsoft/vscode-python/issues/new
2023-07-07 22:38:26.077 [info] Running PYTEST execution for the following test ids: test_example.py::test_example
2023-07-07 22:38:26.077 [info] Server listening on port 35477
2023-07-07 22:38:26.084 [info] Running pytests with arguments: /home/phil/.vscode/extensions/ms-python.python-2023.13.11881007/pythonFiles/vscode_pytest/run_pytest_script.py --rootdir /home/phil/Projects/pytest-vscode-parameters --test_option

2023-07-07 22:38:26.084 [info] > /usr/bin/python ~/.vscode/extensions/ms-python.python-2023.13.11881007/pythonFiles/vscode_pytest/run_pytest_script.py --rootdir . --test_option
2023-07-07 22:38:26.084 [info] cwd: .
2023-07-07 22:38:26.199 [info] Test server connected to a client.
2023-07-07 22:38:26.214 [info] Client disconnected

    And this if I just run the discovery with the Refresh Tests button:

    2023-07-07 22:42:58.794 [info] Discover tests for workspace name: pytest-vscode-parameters - uri: /home/phil/Projects/pytest-vscode-parameters
2023-07-07 22:42:58.794 [info] Running discovery for pytest using the new test adapter.
2023-07-07 22:42:58.808 [info] > /usr/bin/python -m pytest -p vscode_pytest --collect-only . --test_option asd
2023-07-07 22:42:58.808 [info] cwd: .
2023-07-07 22:42:58.912 [info] Test server connected to a client.
2023-07-07 22:42:58.913 [info] Using result resolver for discovery

    The log does not change if I set or unset the logLevel to Trace. It feels like the settings are not applied at all?

    But we can see that the --test_option asd is added properly on discovery, but not for the real test.

  10. eleanorjboyd commented on Jul 7, 2023

    @eleanorjboyd
    Eleanor Boyd (eleanorjboyd)
    on Jul 7, 2023
    Member

    hm yes I can see from 2023-07-07 22:42:58.808 [info] > /usr/bin/python -m pytest -p vscode_pytest --collect-only . --test_option asd it is running the new code, maybe it didn't print that it was on the setting since that log is set on startup of the extension. Either way it is running the new stuff. Thanks for checking I will look into this more!!

  11. philippun1 commented on Jul 22, 2023

    @philippun1
    Philipp (philippun1)
    on Jul 22, 2023
    Author

    There is a workaround for this issue if I use the parameter like this:
    image
    At least it is usable now, but it would be great if the arguments would work in the same way as for pytest:
    pytest --test_option asd and pytest --test_option=asd both work if I call it directly from a terminal.

  12. eleanorjboyd commented on Aug 11, 2023

    @eleanorjboyd
    Eleanor Boyd (eleanorjboyd)
    on Aug 11, 2023
    Member

    Hi! Sorry haven't followed up on this in a minute. On further investigation, this is related to a larger issue around settings which the team will be discussing soon. I will update with information after that.

  13. eleanorjboyd commented on Aug 18, 2023

    @eleanorjboyd
    Eleanor Boyd (eleanorjboyd)
    on Aug 18, 2023
    Member

    I created a proposal with a solution for this problem, would love your input on the above issue! #21845

  14. melMass commented on Sep 3, 2023

    @melMass
    Mel Massadian (melMass)
    on Sep 3, 2023

    I'm not 100% sure it's related but the same happens when using pytest-timeout:

    Run completed, parsing output
Error while running tests: test_successful_command
TypeError: Cannot read properties of null (reading 'testsuites')

Error while running tests:
TypeError: Cannot read properties of null (reading 'testsuites')

  15. nsheaps commented on Sep 20, 2023

    @nsheaps
    Nathan Heaps (nsheaps)
    on Sep 20, 2023

    Hi all, actually just posted in #22032 about a similar issue and just want to call out that I have args, but when I have -n 8 (part of pytest-xdist for distributing across cpu cores) I can't debug the tests with that error. Just removing the flags works fine and I can debug the tests.

    Pytest-xdist explicitly says --pdb is disabled when distributing tests in the docs

  16. eleanorjboyd commented on Oct 6, 2023

    @eleanorjboyd
    Eleanor Boyd (eleanorjboyd)
    on Oct 6, 2023
    Member

    Nathan Heaps (@nsheaps) thank you for your contribution! We will keep this in mind as we work through the proposal!

  17. Eleanor Boyd (eleanorjboyd)
    changed the title [-]Adding pytestArgs gives TypeError: Cannot read properties of null (reading 'testsuites')[/-] [+]pytest arg parsing bug[/+] on Dec 7, 2023

  18. schwaerz commented on Jul 8, 2024

    @schwaerz
    Jörg Schwärzler (schwaerz)
    on Jul 8, 2024

    I am getting this error on a very simple pytest project. Cannot get debugging to work.
    It does not matter for me whether I use pytestArgs or not. Will always get the same error when trying to debug a test:

    Error while running tests: test_git_auth_foo_bar
TypeError: Cannot read properties of null (reading 'testsuites')

Error while running tests:
TypeError: Cannot read properties of null (reading 'testsuites')

    Not sure what's going on.

    Just tried in another project. Same error there :(

    This used to work, as there has been some time while I did not do any python development I cannot say at what time it stopped working.

  19. eleanorjboyd commented on Jul 9, 2024

    @eleanorjboyd
    Eleanor Boyd (eleanorjboyd)
    on Jul 9, 2024
    Member

    hm interesting- could you open an new bug report since this is likely a different problem? Primarily I would need logs from the python output channel but if you use the report issue command on vscode it will gather all this info for you. Thanks!

  20. schwaerz commented on Jul 9, 2024

    @schwaerz
    Jörg Schwärzler (schwaerz)
    on Jul 9, 2024

    Ok, will do that, thanks.

  21. schwaerz commented on Jul 9, 2024

    @schwaerz
    Jörg Schwärzler (schwaerz)
    on Jul 9, 2024

    For reference: #23776

  22. Eleanor Boyd (eleanorjboyd)
    added
    needs spikeLabel for issues that need investigation before they can be worked on.
    on Dec 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area-testingbugIssue identified by VS Code Team member as probable bugneeds spikeLabel for issues that need investigation before they can be worked on.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions