Repository navigation
Test discovery fails if package is not installed #10469
Description
Environment data
- VS Code version: 1.42.1
- Extension version (available under the Extensions sidebar): 2020.2.64397
- OS and version: Ubuntu 18.04
- Python version (& distribution if applicable, e.g. Anaconda): 3.6.7
- Type of virtual environment used (N/A | venv | virtualenv | conda | ...): venv
- Relevant/affected Python packages and their versions: N/A
- Relevant/affected Python-related VS Code extensions and their versions: N/A
- Jedi or Language Server? (i.e. what is
"python.jediEnabled"set to; more info How to update the language server to the latest stable version #3977):false - Value of the
python.languageServersetting:Microsoft
Expected behaviour
When launching vscode, the tests from the tests/ directory should be discovered successfully; regardless of whether the project has been installed as a package.
Actual behaviour
Import errors are thrown out in the output tab and test discovery fails.
Example output:
python <path_to_home>/.vscode/extensions/ms-python.python-2020.2.64397/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir <path_to_project> -s --cache-clear -o junit_family=xunit1
Test Discovery failed:
Error: ============================= test session starts ==============================
platform linux -- Python 3.6.7, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: <path_to_project>
collected 0 items / 4 errors
==================================== ERRORS ====================================
__________________ ERROR collecting <path_to_test_file> ___________________
ImportError while importing test module '<path_to_test_file> '.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
<path_to_test_file>:5: in <module>
import <module_within_project>
E ImportError: cannot import name '<module_name>'
Running pytest separately, with the same launch arguments works fine in a terminal window, but when executed using vscode's test discovery function, it fails to import unless the project is installed as a package first (i.e. by running python setup.py install --record record.txt)
Steps to reproduce:
- Clone this sample repository: https://github.com/RobTheFiveNine/vscode-python-bug
- Create a venv:
virtualenv venv -p python3 - Activate it:
source venv/bin/activate - Install pytest:
pip install pytest - In a terminal, to verify pytest works, run
python -m pytestin the project directory - To verify it doesn't work in vscode, open vscode and configure the Python extension to use pytest, see that test discovery fails
- Open the output tab and view the Python Test Log section to see the errors
- After seeing it fail, run
python setup.py install --record record.txtto install the project as a package in the venv, and then click on the tests label in the status bar or relaunch vscode, to find that vscode now discovers the tests successfully
Logs
python <path_to_home>/.vscode/extensions/ms-python.python-2020.2.64397/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir <path_to_project> -s --cache-clear -o junit_family=xunit1
Test Discovery failed:
Error: ============================= test session starts ==============================
platform linux -- Python 3.6.7, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: <path_to_project>
collected 0 items / 4 errors
==================================== ERRORS ====================================
__________________ ERROR collecting <path_to_test_file> ___________________
ImportError while importing test module '<path_to_test_file> '.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
<path_to_test_file>:5: in <module>
import <module_within_project>
E ImportError: cannot import name '<module_name>'
Activity
brettcannon commentedon Mar 9, 2020
Unfortunately this is a shortcoming/design of pytest as it imports code to do its test discovery and we need pytest to tell us what tests there are and where those tests are. So for us to do this would require pytest updating their test discovery to not require importing code to detect tests.
RobTheFiveNine commentedon Mar 9, 2020
Running pytest on its own doesn't result in the same error, it's only when executed via the vscode extension. For example, if you run
python -m pytest --coin the example directory, the output looks like this:brettcannon commentedon Mar 9, 2020
RobTheFiveNine we will have a look!
kimadeline commentedon Mar 13, 2020
Hi RobTheFiveNine , thank you for reaching out.
I was able to replicate your issue with the sample repo and steps you provided, so thank you for that as well, they were very helpful!
👉 Instead of having to install the project as a package, adding an
__init__.pyfile to thetestsfolder allowed me to run test discovery without issues, let me know if that works for you too.📋 Notes
When running
python ~/.vscode/extensions/ms-python.python-2020.2.64397/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir <path/to>/vscode-python-bug -s --cache-clear tests(the command run by PVSC), the equivalent command (that we display) ispython -m pytest --collect-only --rootdir <path/to>/vscode-python-bug -s --cache-clear tests.Running the PVSC command fails, but running the equivalent command works:
Adding
__init__.pyto the source folder didn't fix test discovery, but adding__init__.pyto the test directory did.Brett Cannon (@brettcannon) something to do about how
run_adapter.pysees the test directory as a top-level package?17 remaining items
mahmoudajawad commentedon Apr 20, 2020
I'm facing the same issue. I have this structure:
So, I have
__init__file across all the path. However, it seemsvscodeis gettingpytestto collect tests at folder level, that's why top-level modules are not available for importing, thus failing. What I need to understand is if there's a way to updatevscodediscovery strategy to depend on regularpytestattempt to run all tests (even if some fails) and then collect the tests from that. Yes, this means tests discovery might take forever because it will run all the tests, but at least this can be a quick hack to test locally until your team comes with better strategy.Another option might be capturing the output of
pytestdiscovery command and listing the tests even if they fail to be imported at discovery time. Am I making sense? I have to ask as I'm not entirely aware of how you are readingpytestor any other test framework output.brettcannon commentedon Apr 21, 2020
Mahmoud Abduljawad (@mahmoudajawad) we aren't reading pytest output, actually. We inject a pytest plug-in that gets run as part of discovery and use that to do the recording of what tests exist.
mahmoudajawad commentedon Apr 22, 2020
Brett Cannon (@brettcannon), I see. That makes sense. I'll try to look on what's going on and see if I can patch it to make it record tests even if pytest fail.
mahmoudajawad commentedon Apr 22, 2020
UPDATE: nvm, I found at it:
Brett Cannon (@brettcannon), can you direct me to where is the plugin that is being injected? I couldn't figure it out myself through the quick scan of the structure being offered in this extension.
brettcannon commentedon Apr 22, 2020
Mahmoud Abduljawad (@mahmoudajawad) the Python code is in https://github.com/microsoft/vscode-python/tree/master/pythonFiles/testing_tools, the TS code is in https://github.com/microsoft/vscode-python/tree/master/src/client/testing/pytest
eleanorjboyd commentedon Jul 14, 2023
dup of #13301 which has just been fixed and is insiders.