Repository navigation
Test dicovery doesn't provide feedback on failure #16786
Description
I find quite often my tests "disappear" from the test window, or discovery appears to do nothing. This is almost always because I've introduced a syntax error in the package I'm testing which results in an import failure. The reason is clear in the Python Test Log, i.e. below I forgot to install a dependency culminating in discovery failing with exit code 2. If this should happen at the very least the test sidebar should provide some indication that test discovery exited abnormally - perhaps with a highlight on the "Show Test Output" icon since clicking on that is the first step in resolving the issue.
Test Discovery failed:
Error: ============================= test session starts ==============================
<snip>
import ahocorasick # pip install pyahocorasick
E ModuleNotFoundError: No module named 'ahocorasick'
=========================== short test summary info ============================
ERROR tests/test_label_model.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
===================== 5 tests collected, 1 error in 2.00s ======================
Traceback (most recent call last):
File "/home/david/.vscode/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
main(tool, cmd, subargs, toolargs)
File "/home/david/.vscode/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
parents, result = run(toolargs, **subargs)
File "/home/david/.vscode/extensions/ms-python.python-2021.7.1060902895/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)
Activity
karthiknadig commentedon Jul 26, 2021
We have major improvements coming to testing UI. See here #16769 this is addressed there.
karthiknadig commentedon Jul 26, 2021
Sorry. I misunderstood. The changes in #16769 provides a way for us to report test discovery errors in a better way. We need to hook this information we get from python to the node where the error occurs in the UI.
aholmes commentedon Aug 3, 2021
To confirm, does this mean the Test UI will not clear out when test discovery fails from, e.g., a syntax error?
karthiknadig commentedon Aug 3, 2021
You will see something like this. We add a node for that workspace with the error details in the test-UI itself.
aholmes commentedon Aug 3, 2021
Does that mean previously discovered tests will not still be displayed?
karthiknadig commentedon Aug 3, 2021
Depends on the test framework, and type of error. On unittest, the test in that particular file with syntax error will be skipped. In pytest, currently syntax errors means it won't be able to see any of the other tests. We have plans on making that part better when we update our integration with pytest.
2 remaining items
eleanorjboyd commentedon Jul 14, 2023
Hello! We have just finished our testing rewrite and are beginning the roll out to users. I have tested this issue with the re-write and I am no longer reproducing the bug! If you would like to try it yourself, you need to add this setting to your users
settings.json"python.experiments.optInto": ["pythonTestAdapter"]. Let me know if it doesn’t work for you and we can re-open this issue. Thanks!aholmes commentedon Jul 14, 2023
Hi Eleanor Boyd (@eleanorjboyd), is there something else we need to do for this setting to take effect?
aholmes commentedon Jul 14, 2023
Actually, it looks like it kind of works after putting the setting in my WSL config.
eleanorjboyd commentedon Jul 14, 2023
You can confirm you have the rewrite enabled by setting
"python.analysis.logLevel": "Trace",in your user settings then check forExperiment 'pythonTestAdapter'is active in your python logs. Do you see that in your logs?aholmes commentedon Jul 14, 2023
Looks like it is.
Here are my observations so far:
aholmes commentedon Jul 14, 2023
By the way, I selected to install the pre-release version of the Python extension. I'm not sure whether that's needed to test the
pythonTestAdapterchange. I was running the release version prior to this.eleanorjboyd commentedon Jul 17, 2023
Hi! Could you send me your logs? I am wondering if there is an error there about UUIDs and if so would be connected to this: #21599