perf(pytest collection): long runtimes for larger number of tests

While there are some other related issues that could help mitigate the pain of long test discovery runtimes (#23047, #4586), I'm writing this issue to ask about potential opportunities to simply improve the performance of the vscode_pytest.

I'm actually not entirely sure where the bottleneck is. Is it in build_test_tree as it iterates over large numbers of tests, or is it in send_message due to the size of the payload? Either way, the performance I'm seeing is worse than I would expect.

Currently
I have ~150,000 test cases, many of which are marked skip or xfail. Manually running python -m pytest --co -q takes 7s, but test discovery executed by vscode takes over 50s.

At some point in the discovery process, the IDE does seem to hang a little bit (not for the entire 50s, but perhaps for rendering the Test Explorer nodes once the message is tests payload is received?).

Expected

I would expect that building the VSCode test tree could occur in an amount of time that's closer to the test collection time when vscode_python plugin isn't involved. The overhead of building the test tree is greater than seems necessary (without an intimate knowledge of everything it's doing).

Workarounds

I know that I can reduce the frequency of test discovery events and that #4586 may also eventually help with this.

Thank you for this issue! I will investigate. It is likely building the test tree in the UI (so on the typescript side) and I have had it in my todo to try and make this more effective.

Eleanor Boyd (@eleanorjboyd)

Thanks for your acknowledgement, Eleanor Boyd (@eleanorjboyd) !

It is likely building the test tree in the UI (so on the typescript side)

While that may be another bottleneck to performance, I think the majority of the duration I'm concerned about is limited to the vscode_pytest plugin because the time difference is apparent in the pytest report itself. Here's that summary from my most recent refresh of the test tree:

================== 169433 tests collected in 77.83s (0:01:17) ==================

Unless... does the vscode_python extension build_test_tree logic only return/exit when it gets a response from the VSCode python extension server which won't come until the UI test tree is fully constructed? That seems pretty unlikely to me.

Eleanor Boyd (@eleanorjboyd)

Eleanor Boyd (@eleanorjboyd) - I'm interested in your thoughts on #25348 (comment) since it seems unexpected to me that the slowness would be on the typescript side given the symptoms.

Eleanor Boyd (@eleanorjboyd)

Eleanor Boyd (@eleanorjboyd) - I just wanted to circle back to provided an update about this experience since I know that several performance improvements have gone into the JS Test Explorer node updater part of the extension. While I believe there's a noticeable improvement in rendering the Test Explorer nodes faster, the performance of the vscode_python extensions pytest_sessionfinish hook is still painfully slow for large numbers of tests. My most recent text indicates an 8x slowdown (15s vs 120s) from the same collection performed manually (without vscode_pytest) vs via the extensions collection command (though I'm not sure if it's linear or not).

I'm quite convinced that the performance inefficiencies lie within that pyest extension's code area (not JS).

vscode-python/python_files/vscode_pytest/__init__.py

Line 373 in 900ae9c

def pytest_sessionfinish(session, exitstatus):

yes I need to review this area- sorry for the delay getting around to it

perf(pytest): cache path resolution to reduce discovery time for larg…

Optimize pytest duplicate check from O(n) to O(1) using sets (#25658)

Tyler BoddySpargo (@tboddyspargo)

Hi Tyler BoddySpargo (@tboddyspargo)! Are you able to run this and see if you see any timing improvements? The update I made should be useful but I could also investigate more if you do not see a substantial difference

Eleanor Boyd (@eleanorjboyd)

Thanks for following up, Eleanor Boyd (@eleanorjboyd) !

Unfortunately, it doesn't appear that there's been much improvement for my suite of tests. Possibly a little bit, but not the order of magnitude improvement I was expecting to see.

Collecting at the command line (without vscode_pytest):

328987 tests collected in 10.43s

Collecting using VSCode Test Explorer (tested with ms-python.python versions 2026.0.0 and 2026.1.2026010901):

328987 tests collected in 66.07s (0:01:06)

Is there any logging/tracing within the build_test_tree logic that I can turn on to help you investigate the source of delay that aren't scaling well to hundreds of thousands of tests?

I can see from #25658 that you suspected the use of list and duplicate checks to be a significant contributor, so I'll point out that there are other such uses ("children" key) that may also require attention. E.g.

vscode-python/python_files/vscode_pytest/__init__.py

Lines 648 to 649 in e2681d5

    
           if test_node not in function_test_node["children"]: 
        
               function_test_node["children"].append(test_node)

However, I'm inclined to suspect that there are also other sources of inefficiency involved.

I'll point out that there are other such uses ("children" key) that may also require attention. E.g.

vscode-python/python_files/vscode_pytest/init.py Lines 648 to 649
 if test_node not in function_test_node["children"]: 
     function_test_node["children"].append(test_node) 
However, I'm inclined to suspect that there are also other sources of inefficiency involved.

Actually, I did some deeper testing with vscode_pytest locally and found that the key inefficiencies all seem to stem from either A) avoiding duplicates in a list (as you suspected) and B) performing redundant computations for every parameterized Item of a single function. Here's what I would suggest (prioritized by the impact on runtime performance):

Continue avoiding inefficient deduplication any time a list is used, esp. for the test cases of a parameterized function (just remove the first duplicate check in process_parameterized_test?).
create_test_node redundantly extracts the line number from every parameterized instance of a test function
- This could be cached by the parent function id rather than the parameterized item ID
Alghough this will be more effort, I think that the payload size could be drastically reduced if you changed the JSON schema sent back to the TEST_RUN_PIPE to avoid duplicating the absolute path so many times.
- If you store the root package folder once and allow all test paths/IDs to remain relative it would save both computation time and payload size (memory and copy/transfer speed).
- If you store path, class+function name, and parameterized ID separately, then many fields of the test nodes themselves will be much smaller.
It might help to use cached_fspath any time that you need os.fspath to optimize for caching.
I would suggest extracting and consolidating the parent/module/file node creation logic that's duplicated for multiple branches of build_test_tree and process_parameterized_test
- The way I see it, the first two conditional branches of build_test_tree should result in the identification of a top-level function/class node (and the creation of all its children class/test nodes). Then, as a separate logical section within the for loop of build_test_tree, the file-level node can be added to file_nodes_dict (if not present) and the top-level item added to its children.
- Once you do that, I think it some additional short-circuiting opportunities will become obvious since many functions will share the same parent module/file.
Generally speaking, I think there may be some unnecessary extraction/computation/conversion of file paths (and possibly some IO cost that could be avoided).

Even doing only 1 and 2 would be a huge improvement.

Eleanor Boyd (@eleanorjboyd)

Eleanor Boyd (@eleanorjboyd) - I wanted to bump my response and analysis from #25348 (comment). I would suggest reopening to address some (i.e. 1 & 2), if not all of those items, but if you'd prefer a new issue, let me know what you think should be in scope for it.

Navigation Menu

Repository navigation

perf(pytest collection): long runtimes for larger number of tests #25348

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if test_node not in function_test_node["children"]:
	function_test_node["children"].append(test_node)

perf(pytest collection): long runtimes for larger number of tests #25348

Description

Activity

eleanorjboyd commented on Aug 4, 2025

tboddyspargo commented on Aug 4, 2025

tboddyspargo commented on Aug 12, 2025

tboddyspargo commented on Oct 17, 2025

eleanorjboyd commented on Dec 12, 2025

eleanorjboyd commented on Jan 6, 2026

tboddyspargo commented on Jan 9, 2026

tboddyspargo commented on Jan 10, 2026

tboddyspargo commented on Mar 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions