Proposal: Enable separate configs per run/discovery/debug/coverage with testing

The following outlines a proposal for how to handle testing args.

Submitted issues related to this problem:

Add separate python.testing.pytestArgs settings for regular, debug and discovery runs #21255
Argument filtering for pytest fails to handle custom arguments #11736
pytest arg parsing bug #21176
Support different options for discovering and running tests with pytest #15148
Read test debug configuration from settings not just launch.json vscode-python-debugger#532
Support setting environmental variables for the test runner #17171
design of UI as someone selects args: Test discovery flow is unusual/weird #17094
launch.json "args" don't get passed to pytest unittests #12997
Add "Run Tests" to launch.json #11932
launch.json "args" don't get passed to pytest unittests #12997
setup testing configs will be changed: Unittest rewrite - configure tests quick pick #21116
people can use plugins on only a single on of the actions: Test Discovery fails on large codebases #21922?
fixing how setup works to be less confusing: Configure Tests command overrides project testpaths #23714
As part of this overhaul we will stop parsing args:
consider coverage as well: Provide separate configurations for running tests with coverage #24865

Current design:
Currently, to pass any command line args for test run/discovery via the extension there is a single setting per test type (one for unittest, one for pytest). The two settings are python.testing.pytestArgs: [] and python.testing.unittestArgs: [] which take a string array of command line args. The args are then processed by the extension to remove some not applicable to the current action (removing run args during discovery and discovery args during run) but this is buggy and hard to maintain. Additionally to debug a debug config can be created in the launch.json.

Issues with current design:

With only a single place to put args for each test type, a user cannot distinguish between which command line args they want to include during test run, discovery, coverage, or debug.
Since pytest does not allow some args to be included during with the arg --collect-only (used to distinguish run from discovery by the extension) these args are parsed out by the extension before sending the request. This means custom args are removed from test args and generally it creates a buggy experience that is hard to maintain with the breadth of args users would like to pass into their tests.
There is no straightforward solution to providing environment variables while running and discovering tests.

Proposed new design:
Create new custom launch configs that can be setup for test run and discovery. Also, increase compatibility with debug launch configs to make sure these work in tandem. The launch config would be set up something like this:

    {
      "name": "Python: pytest run",
      "type": "python-pytest",
      "request": "launch",
      "args": ["", ""],
      "env": {
        "PYTHONPATH": "${workspaceFolder}"
      }
    },
    {
      "name": "Python: pytest discover",
      "type": "python-pytest",
      "request": "launch",
      "args": ["", ""],
      "env": {
        "VAR": "true"
      }
    }

The two particular fields "args" and "env" would provide functionality useful to users. The rest would be default values.

The new launch config types that would be added would be:

Python: pytest run
Python: pytest discover
Python: unittest run
Python: unittest discover
The names of each one are up for discussion, please provide feedback below!

Next a way to set up these launch configs would be introduced to the extension. The initial steps would be:

add configuration -> python -> python config menu
Two possible options for how to display the testing config options would be:
A: to have another submenu called something like "testing" which, when opened, lists the 4 config options above
B: have the 4 config options sit in the current python submenu.

(I think option A is better for clarification even though it requires one more menu clickthrough).

additionally the button "configure tests" found on the test explorer would be changed so it routed the user to a menu of the launch config options for testing (similar to option A).

Concerns with proposed design:

The use of launch configs may be harder for new / beginner users.
The discoverability of launch configs is harder.
The transition from a setting to using a config could be complicated/confusing for current users.

Possible ways to address above concerns

help beginner users
a. Increase documentation and provide specific examples of launch configs.
b. provide default values for all fields in the launch config (this is already planned so the launch config should be runnable upon creation without edits meaning no args or env vars need to be provided)
discoverability
a. There is already a larger discussion happening on how we can improve this, switching to configs for testing provides even more reason to tackle this problem
b. switching the button "configure tests" to the new config flow is very discoverable since that is already the workflow
transition
a. We can have a deprecation notification regarding the setting and show it to all those still using the setting for test args
b. It should be simple to take someone's test args in their settings and turn that into a launch config so we could offer a button that handles this for the user to switch them over (drawback is we would be setting discovery and run to the same args but the user could see this then go change it)

Notes on Debugging:
Less investigation has been done into test debugging as we already offer a similar launch config. Further analysis should be done to see what is missing in this flow and how to improve it to align with test configs once a design is decided on.

Action Items:
We are looking for feedback on this proposal so please comment below any thoughts. Thanks!

The "name" of a launch config is for the name that the user picks, it shouldn't be used to change the behavior of the launch config. Also what does "discovery" involve? We shouldn't use launch configs for anything that isn't basically running/debugging a program.

Another suggestion, I don't know how much complexity is behind the "python-pytest" debug adapter, but if the point is just to invoke the python DA with some args, another option would be to add a config snippet to help users figure out how to write their pytest launch configs. An example of this is the snippets that we have for js-debug to help users write launch configs that start an npm script.

Yes, I forgot to update the "name" of the launch config to represent that as a user input and instead focus on using "type" to change the behavior.

discovery involves running pytest discovery. It is equivalent to running "python -m pytest --collect-only" from the command line. So it is running python code if that is the concern.

I will look into the js-debug snippet! Thanks

adding microsoft/vscode-python-debugger#532 to the related issues as this discusses limitations with current setup.

discovery involves running pytest discovery

Yes- and what is pytest discovery? 😄

and instead focus on using "type" to change the behavior.

I would say this also isn't what I expect- "type" should indicate which debug adapter is being used, and this should just be "pytest" or even "python" or something like that, and then there should be some other argument that changes the behavior.

In that issue, it sounds like it's discussing limitations with launch configs, not settings? I don't understand what Karthik means when he says

We currently only read this from the launch.json. We could expand the scope of where we read it from, but that has some things that we have not decided yet.

in microsoft/vscode-python-debugger#532? I tagged it since resolving this args issue would then remove the setting entirely so it would simplify this process since the launch config would be the only place to reference. I mis-attributed it to "limitations" it is more like this issue would provide clarity to the issue brought up in microsoft/vscode-python-debugger#532.

Yes- and what is pytest discovery? 😄

During discovery pytest goes through and discovers all files that fit the file format for test files and finds tests within those files (both test classes and just functions). It also should read markers to determine which tests to return. So it isn't executing any of the users test code.

I would say this also isn't what I expect- "type" should indicate which debug adapter is being used, and this should just be "pytest" or even "python" or something like that, and then there should be some other argument that changes the behavior.

gotcha yeah that makes sense

During discovery pytest goes through and discovers all files that fit the file format for test files and finds tests within those files (both test classes and just functions)

This doesn't sound like a good match for a debug config then, if I'm not going to be setting breakpoints and debugging my code. It sounds more like a task perhaps? Or just something that the extension does, configured by some settings.

Brett Cannon (@brettcannon)

Brett Cannon (@brettcannon), Karthik Nadig (@karthiknadig), Luciana Abud (@luabud) : thoughts on roblourens comment?

roblourens

replies to roblourens comments:

2.1: If it's in a launch.json, it should be debugging by default 2.2: I don't understand what you mean 2.3/2.4: I know you said

Debug by default makes sense, that solves 2.3 as well. 2.4 was just about duplicating settings so I just more wanted to repeat that question there

but I agree with Connor and if I was doing this I would probably leave out this part of referencing settings.json configs for simplicity. I wonder how complex the configurations typically are? But if this is really important, I think the pattern is fine.

I think its common to have a few environment variables and args but maybe Luciana Abud (@luabud) can speak more to this question.

"purpose" doesn't quite mean the right thing to me. I would suggest "extends", implying that it will use that settings.json config as a base, but you can override properties from the settings.json config by setting them in the launch config as well.

yes this sounds fine to use "extends" if we do want them to build off each other.

2.6: There is a builtin command "Run without debugging", it starts your launch config with the "noDebug" property added dynamically, I would suggest building on that.

this is great ya, no need then for 2.6

Brett Cannon (@brettcannon)

I think this is possible that the configs could have an associated pytest/unittest with each and it runs the given one. So if you select "run profile A", somewhere in the config for "run profile A" you defined it runs with "pytest" so then anytime you attempt "run profile A" it selects "pytest". Brett Cannon (@brettcannon) or Luciana Abud (@luabud) is this too hard for users to do or to discover?

I would expect users to never swap between them once they have chosen their test runner.

For Rob and Connor's edification: pytest can run pytest and unittest testing code (i.e., be a test runner for anyone), but unittest can only run unittest testing code, hence why users even have to specify anything (and this sparks an idea I'm going to go talk to Eleanor about 😁).

Brett Cannon (@brettcannon)

Following this round of comments, here is my new proposal on the design.

settings.json design

{
      "name": "run config: unique name", // user defined name
      "args": ["-v"],
      "env": {
	"PYTHONPATH": "${workspaceFolder}",
        "DJANGO_TESTING": "true",
	"DJANGO_SETUP_SCRIPT_PATH": "/path/to/file",
      },
       "envFile": "path/to/file",
       "testRunProfileKind": ["discovery", "run", "debug"] // all run kind will also display under debug, but not vice versa 
},
"python.testing.framework": ["unittest","pytest"] // this is NOT required, the default will be used if not included

therefore the simplest config if you do all the most basic setup steps with pytest for example would be:

{
      "name": "pytest discovery config", 
      "args": [], // this might even be optional too- open to suggestions on if this is a required part of the json.
       "testRunProfileKind": ["discovery"]
},

default values

default value for testing library (unittest or pytest) will be pytest if it is installed in the environment and unittest if not with this value being configurable via a single setting "python.testing.framework".
the default config will now be defined by the dropdown menu on the Test Explorer tab for run and debug like so:

launch.json design
Launch.json configuration for debugging testing will stay exactly as it is now with the current functionality and design. Users will NOT be able to reference their settings.json test configurations from their config.json file. They will need to duplicate / rewrite any configurations they want to use as a debug config there. This will cause duplication of work but I believe will be the simplest for the MVP and I predict users will tend to stick to a single way of use that will make sense to them. Brett Cannon (@brettcannon), Luciana Abud (@luabud), Karthik Nadig (@karthiknadig) let me know your thoughts on this as I most unsure about this decision and not sure it impact down the road.

Possible UI design:
Will use TestRunProfiles and their related UX. I will update this issue shortly with the flow of how users will set up testing once we switch. Additionally Connor Peet (@connor4312) would it be possible to get a dropdown the same as on the "run" and "debug" icons on the reload icon to allow us to configure different profiles of kind "discovery" that could appear here? With the same option to set a default?

If we can't unify in a single file then I am afraid duplication is just unavoidable. We can operate under the assumption that if you're debugging a test it's a single test and thus going to be a bit specific rather than generic to your whole test suite.

One question about "python.testing.framework" is whether that should be global or should we make a per-config setting (thinking of the monorepo case)?

Eleanor Boyd (@eleanorjboyd)

hi Eleanor Boyd (@eleanorjboyd)! I love this proposal, it would fix an issue my org has with our test suite (we use pytest-xdist to parallelize test runs, but this breaks when we try debugging tests in VS Code). Is there a timeline for when this will be supported?

Is there a timeline for when this will be supported?

Eleanor is busy w/ non-test stuff this month, but the plan is to pick it back up in April.

+1 for this. The collection thing has been really frustrating. Not just that it doesn't work with some args, but also that if you have coverage enabled like so, it overwrites the coverage report. Not critical but it is annoying if you have it setup to discover on save.

The debugging portion of this is a deal breaker. It's working again now, but my issue #22820 is an example of how the current solution is brittle.

"python.testing.pytestArgs": [
        "--cov",
        "--cov-report=html",
        "-vvv",

Any updates on this ? It would be great to be able to use different pytest arguments for single tests, multiple tests, debug test, coverage tests and test discovery.

It is still on our radar but requires larger coordination so we are still in the process. Thanks!

Please also add a way to run pytest discovery with SUDO permissions. Totally not clear how to do that now and several of my test environments require sudo/root permissions. (ie, these modify or access hardware drivers, thus require it)

Notice you may also need to run different tests with different flags for pragmatic purposes: e.g. some tests that require data migrations cannot be ran with --nomigrations while others can be sped up by specifying the flag. You may want to manually choose one or the other.

Navigation Menu

Repository navigation

Proposal: Enable separate configs per run/discovery/debug/coverage with testing #21845

The following outlines a proposal for how to handle testing args.

23 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Proposal: Enable separate configs per run/discovery/debug/coverage with testing #21845

Description

The following outlines a proposal for how to handle testing args.

Activity

github-actions commented on Aug 18, 2023

roblourens commented on Aug 22, 2023

roblourens commented on Aug 22, 2023

eleanorjboyd commented on Aug 22, 2023

eleanorjboyd commented on Aug 23, 2023

roblourens commented on Aug 23, 2023

roblourens commented on Aug 23, 2023

eleanorjboyd commented on Aug 29, 2023

eleanorjboyd commented on Aug 29, 2023

roblourens commented on Aug 29, 2023

eleanorjboyd commented on Aug 29, 2023

23 remaining items

eleanorjboyd commented on Oct 18, 2023

brettcannon commented on Oct 18, 2023

eleanorjboyd commented on Oct 20, 2023

brettcannon commented on Oct 23, 2023

cameronbrill commented on Mar 1, 2024

brettcannon commented on Mar 4, 2024

alenzo-arch commented on Apr 26, 2024

TimotheeJeannin commented on Jan 8, 2025

eleanorjboyd commented on Jan 9, 2025

MicroNateId commented on Feb 25, 2025

sdeleon28 commented on Mar 14, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions