TPI: debugging, multiroot, and large workspace testing with unittest #21306

Refs (overarching issue): #19098

• macOS Benjamin Pasero (@bpasero)
• linux Karthik Nadig (@karthiknadig)
• windows Michael Lively (@Yoyokrazy)
• anyOS Paula (@paulacamargo25)
  Complexity: 4

Author: Eleanor Boyd (@eleanorjboyd)

Create Issue

Thank you for doing this testing! I apologize for how long it is and please let me know if you have any questions!
Prerequisites:

1. Use the following repo of tests or use your own repo of pytests https://github.com/eleanorjboyd/inc_dec_example_repo

Background Information:

This rewrite is for the underlying infrastructure of how python handles testing therefore all behavior visible to the user should be pretty much the same (minus maybe some error logs going in a better place or test names shortened). Therefore throughout this testing, you can reference the current testing in Python to compare if you think something doesn't look right (or you can always post here too!).

Steps:

Part 0: Setup

1. make sure you have the setting "python.experiments.optInto": ["All"], in your user settings json
2. set the log level to trace
3. go to the python output channel and search for pythonTestAdapter and something should come up that looks like this: 2023-05-26 11:12:04.297 [info] Experiment 'pythonTestAdapter' is active (this confirms you are actively in the experiment)
4. Now open the inc_dec_example_repo/inc_dec_example_test_suite  testing repo folder in vscode
5. use the create environment command from the command palette, select venv and whatever python version you want >=3.7
6. in the terminal run python -m pip install pytest
7. from the command palette run python: configure tests select unittest and "." (root directory) and test_*.py

Part 1: Debugging

1. Go to the testing explorer, and confirm all the tests are discovered and nested correctly by file structure.
2. add some break points to the unittest tests and check to make sure these break points are hit
   1. make sure you can do other normal debugging actions (stop, restart, step over, etc)

Part 2: Large repo

1. If you have a very large python repo definitely try it on that repo but if not you can use the provided repo inc_dec_example_repo/inc_dec_example_test_suite
   1. if you are using the provided repo: in the test_subtests.py file, change for i in range(0, 10): to be much larger like for i in range(0, 10000): and make sure it both works for discovery and run tests (the subtests should not be displayed in the test explorer after discovery only after run)
   2. if you are using your own: make sure all tests can be discovered and run as expected within your large repo when set to use unittests

Part 3: Multiroot Workspace

1. now you are going to open inc_dec_example_repo/folder3 but open it as a multiroot workspace (open the file testing3/testing3.code-workspace)
2. go to the testing panel, make sure that the tests are set up with the correct setting per workspace. This means that testFolder1 and testFolder3 are pytest and testFolder2 is unittest and they should all be displayed the correct tests based on this type.