Support an output stream from the testing adapter script.

Currently the pytest adapter captures the pytest stdout (and hides it unless there's an error) and write the JSON results to stdout. The extension then parses this output and incorporates the results.

An alternate approach would be to connect a socket to the script and push results out over that socket. This would add complexity, but provides a number of benefits:

get results in real-time (important with large projects)
can stream out individual errors instead of failing entirely
no need to hide pytest stdout
does not interfere with normal use of stdout (e.g. see atexit registered functions that print to stdout break test discovery #6581)
get partial results even if the script fails

Bill Schnurr (@bschnurr)

Bill Schnurr (@bschnurr)

Don Jayamanne (@DonJayamanne), we talked about this a while back, but I couldn't find any issue about it...

What you've suggested is exactly what we do with unit tests in Python. and I agree with the benefits. It's a more structured and easier solution.
One change I'd make is to invoke our code as a plugin. I.e. use python -m pytest and ensure it code is treated as a plugin.

This was way we run tests exactly the same way the users do. (Instead of us launching pytest using the API, modifying paths, args, etc)

Does this mean that pytest in vscode currently captures stdout and not show it in the test runner?

No, it is displayed in the test runner.

Thanks.. we are seeing similar issues in VS. For now i'm making a change to optionally write results to a file so that we can still get partial results.

I have updated VSCode, but the test discovering still fails on all my projects (#8123). I'm quite surprised that the issue hasn't been resolved. Can I fix it somehow temporarily? Do you need more information on the issue? Here is my logs:

Error Python Extension: 2019-11-10 16:06:21: Python Extension: displayDiscoverStatus [Error: 2019-11-10 16:06:15.731180: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA2019-11-10 16:06:15.744855: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fc9be6345c0 executing computations on platform Host. Devices:2019-11-10 16:06:15.744904: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version2019-11-10 16:06:16.534371: W tensorflow/python/util/util.cc:299] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.	at ChildProcess.<anonymous> (/Users/artemav/.vscode/extensions/ms-python.python-2019.10.44104/out/client/extension.js:9:37710)	at Object.onceWrapper (events.js:288:20)	at ChildProcess.emit (events.js:200:13)	at maybeClose (internal/child_process.js:1021:16)	at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)]
t.log @ console.ts:137
$logExtensionHostMessage @ mainThreadConsole.ts:39
_doInvokeHandler @ rpcProtocol.ts:398
_invokeHandler @ rpcProtocol.ts:383
_receiveRequest @ rpcProtocol.ts:299
_receiveOneMessage @ rpcProtocol.ts:226
(anonymous) @ rpcProtocol.ts:101
fire @ event.ts:580
fire @ ipc.net.ts:453
_receiveMessage @ ipc.net.ts:733
(anonymous) @ ipc.net.ts:592
fire @ event.ts:580
acceptChunk @ ipc.net.ts:239
(anonymous) @ ipc.net.ts:200
t @ ipc.net.ts:28
emit @ events.js:200
addChunk @ _stream_readable.js:294
readableAddChunk @ _stream_readable.js:275
Readable.push @ _stream_readable.js:210
onStreamRead @ internal/stream_base_commons.js:166

Luciana Abud (@luabud)

Luciana Abud (@luabud), any ideas on timeline?

We're hoping to get to it soon but no ETAs yet.

Luciana Abud (@luabud)

Luciana Abud (@luabud) , is there a workaround? All python extension features for testing in VSCode literally don't work at the moment. Should I just install another extension (if there is any) and forget about vscode-python :)

Artem Artemev (@awav)

Artem Artemev (@awav) I'm so sorry for not giving workaround options 😞

According to what you said here: #8123, the error that is being thrown is this:
Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA

If you suppress the warning, test discovery won't fail. So what you can do is create a .env file with the following content:
TF_CPP_MIN_LOG_LEVEL='2'.

Then open the settings.json file under the .vscode folder in your workspace and add the setting below:
"python.envFile": "${workspaceFolder}/.env"

Reload the window and you should be able to see your tests in the test explorer:

Hello! We have just finished our testing rewrite and are beginning the rollout to users. If you would like to try it yourself, you need to be on vscode insiders and then add this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"]. We are in the process of switching all users to the rewrite but are doing so incrementally so if you do not have insiders, watch our release notes to get updated on when it will begin to hit stable. Closing this issue with this in mind but let me know if it doesn’t work for you and we can re-open this issue. Thanks!

Stoked about this. Trying it out and every time I run my test I see a list of ALL the tests. E.g.

plugins: html-3.2.0, metadata-3.0.0, mock-3.8.2, anyio-3.6.2
collected 261 items

<Package tests>
  <Module test_activity.py>
    <Function test_search_activities>
    ....

This makes it hard to find the output, especially because any time I change a file, it reruns and obscures the logs. Any setting or anything I can change on my side?

Dan Diephouse (@dandiep)

Dan Diephouse (@dandiep) glad it worked! Have you tried adding -q to your pytest settings? This is should reduce noisy output and wondering if that would be the fix. The output that comes in the testing output log is all from pytest so we have limited control over it.

Navigation Menu

Repository navigation

Support an output stream from the testing adapter script. #6594

65 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support an output stream from the testing adapter script. #6594

Description

Activity

ericsnowcurrently commented on Jul 15, 2019

DonJayamanne commented on Jul 18, 2019

domoritz commented on Aug 4, 2019

DonJayamanne commented on Aug 5, 2019

bschnurr commented on Aug 6, 2019

awav commented on Nov 10, 2019

ericsnowcurrently commented on Nov 11, 2019

luabud commented on Nov 13, 2019

awav commented on Nov 19, 2019

luabud commented on Nov 19, 2019

65 remaining items

eleanorjboyd commented on Jul 5, 2023

dandiep commented on Jul 16, 2023

eleanorjboyd commented on Jul 17, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions