Repository navigation
Test failure should not be reported as a compile error #9640
Description
Have a test failing.
Notice that python reports this as an Error. This shows up in the Problems panel and with a squigle.
This is not the right thing to do. Test errors should not be reported as compile errors.
I am not sure if there is a better alternative to use
fyi Johannes Rieken (@jrieken) Luciana Abud (@luabud)
Also we now support rendering emojis so test failures could be rendered a bit better. Johannes Rieken (@jrieken) can provide more details.
Activity
luabud commentedon Jan 17, 2020
FWIW this was implemented by an external contributor for a feature request: #120, PR: #3303.
If I'm not mistaken the Problems window was used because there isn't a way to jump to the failed lines through the Output window?! But I may be wrong.
Anyway, we should then look into a better solution to solve the problem other than using the Problems window. As you pointed out, this isn't a syntax or compilation error.
DonJayamanne commentedon Jan 19, 2020
IMHO I think the problems window is fine.
Though reporting test issues in problems window while we have a test explorer (dedicated UI for tests) does seem awkward.
The user requirement is that when tests fail, they expect to see the failures and be able to jump to the failed lines easily. Currently this isn't possible from the test explorer.
I.e. the test explorer doesn't provide a view for failures in a list. Agreed we can provide a custom view. But users wouldn't know about such a toggle, or we'd need to create another UI list view with exact same functionality as problems window, however that doesn't seem right.
Not sure this is correct. Today we have spell checkers reporting errors in problems window and they aren't compile errors. I wouldn't be surprised if there were others.
That may have been the intent, however I believe the community is now using the problems window to surface other types of problems.
luabud commentedon Feb 26, 2020
Isidor Nikolic (@isidorn) I talked to the team about it and the sentiment was that we always thought having that report in the problems window would be fine (after all, having failed tests in your project could be seen as a problem).
Are there any guidelines from VS Code that we could refer to in terms of what should or shouldn't be displayed in the Problems window?
I think Don raised a good point that the community may be using the Problems window to surface various types of problems, so having some guidelines could help on this.
isidorn commentedon Feb 27, 2020
Adding Johannes Rieken (@jrieken) and Sandeep Somavarapu (@sandy081) for Prolbems view guidelines
sandy081 commentedon Feb 27, 2020
Problems view is for code diagnostics which are generated by a compiler and Tests view is for managing and running tests. Former is compile time and later is during run time. I think this is the standard in general everywhere. As always, we never enforced this but it is expected that extensions follow this so that users will get consistent experience.
Agreed that before having tests view, authors were using different approaches (output, problems, code-decorations) to show tests. But after introducing this, we recommended all to create a test view and show test results there. You can define your own actions to run, debug the tests and more which you cannot do from Problems view where you can only do quick fixes (which is to fix your code to compile).
I do not think I agree with this and they are compile time errors.
jrieken commentedon Feb 27, 2020
The problems view is for syntax or semantic errors and for lint'ing messages. You can obviously bend the definition of a "problem" but when we design and plan features for diagnostics then we do not think about test running, e.g "Run & Debug" will prompt you in the presence of diagnostics - that makes sense for compile errors but not for test errors, esp. not when "run" means "re-run the tests". That means there is a chance of future (semantic) breakage and for non-python users this might be confusing
Still, we like the idea of showing test failures in-line with sources (we just don't like the use of diagnostics for this) and this issue should also be an invitation to brain storming how VS Code (as a platform) should offer better test reporting UX and API. The Jest extension, while also using diagnostics 😞, uses line-end decorations which I think is pretty cool.
So, lets think further than the test explorer and lets start to collect requirements and ideas for better test integration
karthiknadig commentedon Jul 29, 2021
Fixed via #16769