Repository navigation
Custom Test Discovery Option #23047
Description
Type: Feature Request
Behaviour
Expected vs. Actual
Hi there!
So right now, the python test suite discovery relies on pytest running with the --collect-only flag. For normal projects, this works fine as this pytest command would take 1-2 seconds. But if you happen to have a really big python project with 80K tests and counting like in my company, Rippling, then the test collection takes over 10 minutes, and usually breaks the editor with too much data.
Here for the typescript version
And here for the python version
We really need to overcome this if we are to officially adopt VS Code as our editor for the company.
Before making a whole private extension to support this, I would like to double check one more time (I first checked in October, but I know things might have changed since then) if there is anything official coming up for this. This problem is really like the last big blocker, and I'm sure that other companies with large codebases are facing it right now too.
Steps to reproduce:
- Get a codebase with ~80K tests and ~300 dependencies.
- Setup the venv everything usual there.
- Do the regular flow to configure test runner with pytest.
- Watch as your editor stalls for 10 minutes and then enters a broken state.
Diagnostic data
- Python version (& distribution if applicable, e.g. Anaconda): 3.10.11
- Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Poetry
- Value of the
python.languageServersetting: Default
Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)
XXX
User Settings
languageServer: "Pylance"
testing
• pytestArgs: "<placeholder>"
• pytestEnabled: true
experiments
• optInto: ["pythonTestAdapter"]
Extension version: 2024.2.1
VS Code version: Code 1.87.0 (Universal) (019f4d1419fbc8219a181fab7892ebccf7ee29a2, 2024-02-27T23:42:56.944Z)
OS version: Darwin arm64 23.3.0
Modes:
Activity
drothlis commentedon Mar 12, 2024
In #21922 (comment) Brett Cannon (@brettcannon) wrote:
Yes, it does. We wrote a pytest plug-in that does custom collection and execution:
https://david.rothlis.net/running-tests-on-remote-hardware-from-your-ide/
(I'm replying here because #21922 is locked to new comments.)
drothlis commentedon Mar 12, 2024
Juan Diego Palomino (@juandiegopalomino) but do be aware that VSCode's Python plugin occasionally breaks our use of pytest, e.g. #8242/#8425 (our workaround: stb-tester/stbt-rig@89f7a27) and #22076/#22885/#22999 (our workaround: stb-tester/stbt-rig@a88df29). I suppose twice in 5 years isn't too bad, and in both cases we were able to implement a work-around, but it did break the VSCode integration for our users.
The vscode-python team also don't seem to test against older (but still supported) versions of pytest? e.g. #22457. But this one didn't affect us.
I'm just saying you might want to consider writing a custom VSCode plugin; maybe VSCode's plugin API is more stable (but I have no experience with that).
juandiegopalomino commentedon Mar 12, 2024
David Röthlisberger (@drothlis) hi!
Wow! Your code is awesome, it helps so much! One question though-- it looks like the pytest plugin will still execute for all the python files in the repo (i.e. it will go one by one through all of my files). That might still be too long, as that would still be thousands of files 😅 . Is there anyway to say "just run this on the currently opened files?"?
drothlis commentedon Mar 12, 2024
It's just grepping for
^def\s+(test_[a-zA-Z0-9_]*)so it should be fast, even for thousands of files. I suppose it could end up reporting more tests than VSCode can handle?You could try adding
"${file}"to the "python.testing.pytestArgs" in.vscode/settings.json(see Predefined variables) but I don't know if that's supported in pytestArgs (see Is variable substitution supported in User and Workspace settings?).brettcannon commentedon Mar 12, 2024
We do now:
vscode-python/.github/workflows/pr-check.yml
Line 98 in 209d6bd
eleanorjboyd commentedon Mar 27, 2024
hi Juan Diego Palomino (@juandiegopalomino), looks like you got some awesome suggestions from David Röthlisberger (@drothlis), thanks David Röthlisberger (@drothlis) for helping out! Two points:
vltr commentedon Mar 29, 2024
Eleanor Boyd (@eleanorjboyd) I'm going to pry a bit in here, but option two seems very interesting! 😅
eleanorjboyd commentedon Mar 29, 2024
Hi! If you are interested definitely give the original issue an upvote as we decide prioritization of tasks based on community interest (as measured by upvote count) #4586
github-actions commentedon Jun 7, 2024
Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.
Happy Coding!
jbezuk commentedon Mar 5, 2025
Hi, I would love to see this too. My use case is that I have a script/plugin which I can use to fetch a list of failing tests from a remote CI server and execute them in a local test environment with pytest. As the failing tests go from failing to passing, they are saved in a new current failing test list file so I can systematically work through a list of test failures in my local test environment. It would be great if I could have vscode use this as its test discovery/execution engine so I have the normal testing panel UI to view the results.