Testing reliability & perf 2026 — workstream #25978

Tracking the user-reported testing regressions and performance complaints that emerged after the project-structure / [test-by-project] rework, plus the older follow-ups that ride alongside them.

The work is grouped into four problem areas. Each item below points at the existing user-filed issue (or a new engineering issue where there is no good match) so progress is visible in one place.

Problem areas (what users are reporting)

1. Test discovery is much slower than CLI pytest.
   Multiple users on large parametrized suites — 30k tests take 40s in the Test Explorer vs ~2s on the CLI; 328k tests take 66s vs ~10s. The O(n²) list scans and oversized discovery payload have been addressed; follow-up work remains on line-number/path compaction and validating the end-to-end improvement.

2. Test tree is rebuilt from scratch on every change.
   Saving any .py file re-discovers the whole workspace and wipes the existing tree. While re-discovery is in flight, users can't re-run or debug a test because the items have been cleared. "Debug: Restart" breaks for the same reason.

3. Run / debug pipeline regressions.
   Tests appear as "skipped" even though they ran; debug runs can lose results when the result pipe closes; pytest-subtests failures get reported as success; the env selected via the Python Environments API is not always honored.

4. Hard discovery failures still open.
   Smaller correctness bugs in the pytest plugin (HIDDEN_PARAM, pipe writer broken by mock.patch("builtins.open")).

Order of execution (impact × effort)

• Narrow python.testing.autoTestDiscoverOnSavePattern default so saving a non-test file does not trigger a full re-discovery. Default today is **/*.py; change it to conventional pytest/unittest filenames plus conftest.py, while preserving custom user patterns. Implementation and regression tests are complete locally; publication is pending. — python.testing.autoTestDiscoverOnSaveEnabled is running test discovery on change to any file (not just test files) #25866
• Replace list-membership dedup with dict/set in vscode_pytest. Removed the O(n²) scans that dominate heavily parametrized suites. — Extremely slow test discovery when using pytest for large test suite #25973, completed by Improve performance of pytest discovery for large test suites #25974
• Verify result-pipe drainage on cancellation. Implementing result pipe drainage on cancellation #25981 changed cancellation to allow buffered results to drain, but PyTest exits when subprocess finishes execution when ran through Testing UI #25872 remains open and still needs end-to-end verification against the reported debug/subprocess scenario.
• Surface an error payload when the env-extension subprocess exits non-zero. Legacy path already does this in pytestExecutionAdapter.ts; the env-extension path resolves the deferred silently. With no results and no error, every test defaults to skipped. — Python skip all Tests #25892
• Aggregate pytest-subtests reports before publishing the final test outcome. Multiple call-phase reports share one nodeid; the current collected_tests_so_far dedup makes the first report win, so a later failing subtest or final failure can be discarded. — failing pytest tests are marked as “success” when some of the subtests succeeded #25824
• Slim the pytest discovery JSON payload. Store shared path prefixes once and compact repeated absolute paths in test nodes. — [PERF] Pytest Discovery - highly parameterized tests lead to inefficient test node creation in vscode_pytest #25948, implemented by Enhance test discovery payload handling with compact/expansion on paths in payload #25982 with follow-up testing: refine compact discovery payload review feedback #26000. Remaining optimizations are tracked in open PRs optimize compact_path() in pytest discovery for the common case #26016 and optimize getting of line number in pytest discovery #26017; both currently require corrections before merge.
• Incremental tree updates in populateTestTree. Today processDiscovery does testItemIndex.clear() and populateTestTree always rebuilds. Diff old vs new test trees and only insert / remove / update changed items. Largest item; biggest user-visible fix; should land last. — After latest update: Testing tree of python tests reloads completely rather than just update #25822
• pytest.HIDDEN_PARAM discovery crash. process_parameterized_test does parent_part, parameterized_section = test_node["name"].split("[", 1), which raises ValueError when pytest emits a node id without [...] (i.e. when HIDDEN_PARAM is used). One-line guard. — pytest.HIDDEN_PARAM Causes Discovery Failure #25795
• Telemetry step 2 (Python-side). Add a meta block to the vscode_pytest discovery payload with subprocessDurationMs, pluginDurationMs, payloadBytes, parametrizedTestCount, and fold it into the new UNITTEST.DISCOVERY.DONE event so we can split slow discoveries into "subprocess vs plugin vs JS overhead". — new issue
• Make vscode_pytest pipe writer immune to mocked open. When user test code does mock.patch("builtins.open"), the pipe-writer open() call gets intercepted and serialization breaks (surfaces as unsupported operand type(s) for +=: 'int' and 'NoneType'). Capture open at import time before any test code can monkeypatch it. [vscode-pytest]: unsupported operand type(s) for +=: 'int' and 'NoneType' #25793 was closed as not planned after the information-request timeout; no fix landed. — new issue or reopen [vscode-pytest]: unsupported operand type(s) for +=: 'int' and 'NoneType' #25793
• Verify useEnvExtension() actually fires in VS Code 1.106+. Reporter pins a regression to that VS Code release. Telemetry-first: add an envSource field to the discovery/run events, look at the live data, then fix. — pytest discovery does not use specified environment #25718

Closing as already resolved

• Unittest test discovery fails for Python 3.15 #25802 — Unittest discovery on Python 3.15. User confirmed fixed in extension 2026.4.0 on May 9.
• Unwanted debug output when running unittests (again). #25807 — Unwanted unittest debug output. The offending print has already been removed from python_files/unittestadapter/execution.py.

How we measure progress

Instrumentation status:

• Add TS-side discovery/run timing and outcome telemetry. — Add testing performance telemetry #25980
• Send numeric timing/count fields through the telemetry measures channel so they are ingested. — Fix testing perf telemetry: send measurements via measures arg #26028
• Add Python-side phase timing and payload-size/count metadata.
• Document and run the analysis queries/dashboard slices used to validate each fix.

Each fix above has a corresponding metric so dashboards can verify the change actually moves the needle (and catch unintended regressions):

Per-area success criteria are checked off as each fix ships and the telemetry confirms the change.

Out of scope (deliberately, for now)

• Per-test or per-file names in telemetry — privacy-sensitive, not needed for the questions above.
• True added / removed counts in UNITTEST.TREE.UPDATE — needs an O(n) set diff per discovery; revisit only if beforeCount/afterCount + rebuiltFromScratch aren't enough signal.
• Migrating off named pipes entirely — out of scope for this workstream.