Argument filtering for pytest fails to handle custom arguments

Issue Type: Bug

I created a fixture in conftest.py that reads at input args to pytest (using pytest_addoption. See code below. When I try and add this option to my settings.json file so that my tests execute properly, it fails with the ERROR: file not found...

Here is my conftest.py file :

import pytest
import os


def pytest_addoption(parser):
    parser.addoption("--config_file", action="store")


@pytest.fixture(scope='session')
def config_file(request):
    config_file_value = request.config.option.config_file
    if config_file_value is None:
        print(f'config_file_value = {config_file_value}')
        pytest.skip()
    elif not os.path.exists(config_file_value):
        print(f'config_file_value = {config_file_value}')
        pytest.skip()
    return config_file_value

And then my settings.json relevant to this:

{
    "python.pythonPath": "/usr/bin/python3",
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.testing.cwd": "",
    "python.autoComplete.addBrackets": true,
    "python.testing.pytestArgs": [
        "--config_file app/config.json",
    ]
}

And finally the output of the console :

ython /root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /root/jetson-task-manager -s --cache-clear --config_file app/config.json
Test Discovery failed: 
Error: ============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.4.2, py-1.5.2, pluggy-0.13.1
rootdir: /root/jetson-task-manager
plugins: forked-0.2, xdist-1.22.1
collected 0 items

============================ no tests ran in 0.06s =============================
ERROR: file not found: --config_file app/config.json


Traceback (most recent call last):
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 4)

Running pytest version 5.4.2. Feels to me like this option is taken as if it were a file to test, versus an actual argument. Ideas? Am I missing something?

Extension version: 2020.4.76186
VS Code version: Code - Insiders 1.46.0-insider (d487078, 2020-05-07T16:19:54.327Z)
OS version: Windows_NT x64 10.0.18363
Remote OS version: Linux arm64 4.9.140-tegra
Remote OS version: Linux arm64 4.9.140-tegra

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz (8 x 2112)
GPU Status	2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on oop_rasterization: disabled_off protected_video_decode: enabled rasterization: enabled skia_renderer: disabled_off_ok video_decode: enabled viz_display_compositor: enabled_on viz_hit_test_surface_layer: disabled_off_ok webgl: enabled webgl2: enabled
Load (avg)	undefined
Memory (System)	15.93GB (7.08GB free)
Process Argv
Screen Reader	no
VM	0%

Item	Value
Remote	SSH: jetson
OS	Linux arm64 4.9.140-tegra
CPUs	unknown (6 x 345)
Memory (System)	7.68GB (0.66GB free)
VM	0%

Item	Value
Remote	Container taskmanager:latest (/taskmana…
OS	Linux arm64 4.9.140-tegra
CPUs	unknown (6 x 1420)
Memory (System)	7.68GB (0.66GB free)
VM	0%

Olivier Martin (@omartin2010)

Olivier Martin (@omartin2010) You have to split these into separate arguments. Right now it is treating --config_file app/config.json as a filename.

"python.testing.pytestArgs": [
        "--config_file", 
        "app/config.json",
    ]

if I do that, it doesn't work, I get this :

python /root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /root/jetson-task-manager -s --cache-clear --config_file app/config.json
Test Discovery failed: 
Error: ERROR: usage: run_adapter.py [options] [file_or_dir] [file_or_dir] [...]
run_adapter.py: error: unrecognized arguments: --config_file
  inifile: None
  rootdir: /root/jetson-task-manager


Traceback (most recent call last):
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 4)

Turns out I also have another argument that needs to be there which is the directory that contains the tests, so my args are :

    "python.testing.pytestArgs": [
        // "test",
        "--config_file",
        "app/config.json"
    ]

With or without the test argument (not having this line at all or putting it last), I get the same error..

Looking at pytest's help i don't see a config-file but i see a -c. I don't think they have a expanded config file option.

> py -3.8 -m pytest --help | findstr config
                        configuration file raise errors.
  -c file               load configuration from `file` instead of trying to
                        locate one of the implicit configuration files.
test session debugging and configuration:
  -h, --help            show help message and configuration info
  --trace-config        trace considerations of conftest.py files.

Try this:

"python.testing.pytestArgs": [
        "-c",
        "app/config.json"
    ]

I think I may have been unclear. This is for my own, user defined parameter, one one for pytest itself, so that a fixture can retrieve it in this kind of a function. Here is an extract of my conftest.py file:

def pytest_addoption(parser):
    parser.addoption("--config_file", action="store")

so that it's passed to my pytest session. So I want this param to pass straight to my stuff. So far it works when I run this from the command line, just not when from vscode...

Looks like the bug is in filterArguments for pytest:

vscode-python/src/client/testing/pytest/services/discoveryService.ts

Line 46 in 61b179b

    
           const args = this.argsService.filterArguments(options.args, TestFilter.discovery);

We pass the config_file arguments there and we don't get back the expected arguments. In the case of :

        "python.testing.pytestArgs": [
            "--config_file",
            "app/config.json",
            "tests"
        ]

Expected:

python c:\GIT\issues\s p\vscode-python\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir c:\GIT\issues\issue11736 -s --cache-clear --config_file app/config.json tests

Actual:

python c:\GIT\issues\s p\vscode-python\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir c:\GIT\issues\issue11736 -s --cache-clear --config_file app/config.json
python c:\GIT\issues\s p\vscode-python\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir c:\GIT\issues\issue11736 -s --cache-clear --config_file tests

Not sure why it prints out the command twice with different arguments.

Having a very similar problem:
"python.testing.pytestArgs": [
"-rP",
"--html=c:\Data\pytest-report.html",
"--self-contained-html"
],

These are all ignored - not matter what I do with them, they do not effect running tests either with "run test" or "debug test"

window 10, vs code 1.58.2
pytest, pytest-html (among others)

same issue

Karthik Nadig (@karthiknadig)

Karthik Nadig (@karthiknadig) any movement on this issue. It cropped up in regards with another issue as well

We are in the process of re-writing the testing adapter #17242 . It is actively being worked on in the current milestone.

Karthik Nadig (@karthiknadig)

Karthik Nadig (@karthiknadig) - as a fellow open-sourcer - happy to play guinea pig 😉 - thanks for your work

Until a fix available, you can add these options on pytest.ini file,

[pytest]
addopts = --config_file app/config.json

https://docs.pytest.org/en/6.2.x/customize.html

So basically I can't dynamically choose between two different implementations like:
python -m pytest --algorithm merge
python -m pytest --algorithm quick
To reuse the same tests?

Has there been any movement on this? I can't get any of the suggestions or workarounds to work (VS Code Version: 1.80.2)

I created a proposal with a solution for this problem, would love your input on the above issue! #21845

I was able to use this format to get around it:

{
    "python.testing.pytestArgs": [
        "--mycustom-arg1=value1",
        "--mycustom-arg2=value2"
     ]
}

Navigation Menu

Repository navigation

Argument filtering for pytest fails to handle custom arguments #11736

3 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Argument filtering for pytest fails to handle custom arguments #11736

Description

Activity

karthiknadig commented on May 11, 2020

omartin2010 commented on May 11, 2020

karthiknadig commented on May 11, 2020

omartin2010 commented on May 11, 2020

karthiknadig commented on May 18, 2020

3 remaining items

gravity-rose commented on Jul 21, 2021

wkerzendorf commented on Feb 11, 2022

wkerzendorf commented on Feb 14, 2022

karthiknadig commented on Feb 14, 2022

wkerzendorf commented on Feb 14, 2022

binishkaspar commented on Feb 23, 2022

marq4 commented on Jun 10, 2022

PaulWaldo commented on Jul 31, 2023

eleanorjboyd commented on Aug 18, 2023

elongstreet88 commented on May 7, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions