Repository navigation
PyTest optional args don't support wildcards #19918
Description
Type: Bug
Behaviour
Expected vs. Actual
Expected: pytest's positional arguments support wildcards to specify what test directories/files you want to run. E.g. pytest mydir/**/*_test.py.
Actual: Something about the VSCode-Python extension interprets these wildcards literally and results in an ERROR: file not found "mydir/**/*_test.py"
Steps to reproduce:
- Add wildcard file into optional args for Python>PyTest
- Refresh tests in the Testing pane
- Watch as pytest discovery fails because the harness can't find the literal file name (i.e. doesn't interpret the wildcard characters correctly)
Diagnostic data
- Python version (& distribution if applicable, e.g. Anaconda): 3.8.6
- Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Conda
- Value of the
python.languageServersetting: Default
Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)
> ~/miniconda3/bin/conda run -n myenv --no-capture-output python ~/.vscode/extensions/ms-python.python-2022.15.12711056/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.15.12711056/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear --continue-on-collection-errors platform/**/test_*.py
cwd: .
[ERROR 2022-8-28 16:31:18.838]: Error discovering pytest tests:
[n [Error]: ============================= test session starts ==============================
platform darwin -- Python 3.8.6, pytest-7.1.2, pluggy-1.0.0
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /Users/cschlager/work/src2, configfile: setup.cfg
plugins: ddtrace-1.3.0, anyio-3.6.1, benchmark-3.4.1, env-0.6.2, timestamper-0.0.9, mock-3.8.2, hypothesis-6.54.4, hydra-core-1.2.0, cov-3.0.0
collected 0 items
========================= no tests collected in 0.01s ==========================
ERROR: file or directory not found: platform/**/test_*.py
User Settings
languageServer: "Pylance"
linting
• flake8Enabled: true
• mypyEnabled: true
formatting
• provider: "black"
testing
• cwd: "<placeholder>"
• pytestArgs: "<placeholder>"
• pytestEnabled: true
• unittestArgs: "<placeholder>"
Extension version: 2022.15.12711056
VS Code version: Code 1.71.2 (Universal) (74b1f979648cc44d385a2286793c226e611f59e7, 2022-09-14T21:05:37.721Z)
OS version: Darwin arm64 21.6.0
Modes:
Sandboxed: No
System Info
| Item | Value |
|---|---|
| CPUs | Apple M1 Pro (10 x 24) |
| GPU Status | 2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled metal: disabled_off multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_renderer: enabled_on video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off |
| Load (avg) | 4, 5, 5 |
| Memory (System) | 32.00GB (0.11GB free) |
| Process Argv | --crash-reporter-id 58741217-6e7d-4429-9190-74aeb72b3dd3 |
| Screen Reader | no |
| VM | 0% |
A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383:30185418
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
pythonvspyl392:30443607
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vsdfh931cf:30280410
vshan820:30294714
vstes263:30335439
pythondataviewer:30285071
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411cf:30557515
vsaa593cf:30376535
pythonvs932:30410667
cppdebug:30492333
vscaat:30438848
vsclangdc:30486549
c4g48928:30535728
dsvsc012cf:30540253
azure-dev_surveyone:30548225
2144e591:30553903
pyindex848cf:30577861
fc301958:30573243
Activity
eleanorjboyd commentedon Oct 10, 2022
Hello, we are aware of buggy behavior on pytest discovery and running in vscode. We are currently work on a rewrite of how pytest works in vscode and hope this rewrite will fix this and similar bugs. The main issue which will be working on is here: #17242, and progress will be updated on this issue. Thank you!
eleanorjboyd commentedon Aug 18, 2023
Hi! Currently we run pytest discovery using ChildProcess spawn function from nodejs. The spawn function handles the args as an array of strings and therefore does not expand wildcards in args. I think for this reason this is currently not possible with the extension.
Karthik Nadig (@karthiknadig), would this be feasible at some point? Looked and it seems hard to switch from
spawntoexecwhich is the recommended way around it. Also saw thatpytest.maindoes not expand wildcards (but would need to check with maintainers on this) so unsure if we could switch to calling a script that then calledpytest.main.eleanorjboyd commentedon Aug 18, 2023
opened pytest-dev/pytest#11319 to confirm
eleanorjboyd commentedon Aug 18, 2023
Yes it is true they don't support it on pytest.main but I was reminded that we could handle it via glob from the stdlib to expand before passing it in.
Karthik Nadig (@karthiknadig) thoughts on if this would be worthwhile/doable?
eleanorjboyd commentedon Oct 30, 2023
Karthik Nadig (@karthiknadig), following up on this. Confirming it is not something we want to work on right?
karthiknadig commentedon Oct 30, 2023
This is now handled via test explorer. If you want to run a subset of the test, you can use the search field in the test explorer and select the test or test files that you want to run.
That is correct. PyTest doesn't provide us with an API to do this. Given that this is shell dependent, and can get really complex to support generally,
schlagercollin commentedon Oct 31, 2023
Thanks Eleanor Boyd (@eleanorjboyd) and Karthik Nadig (@karthiknadig) for following up on this.