TPI: Dynamic Run Pytest #21494

Refs: #21148

• macOS João Moreno (@joaomoreno)
• linux Raymond Zhao (@rzhao271)
• windows Bhavya U (@bhavyaus)

Complexity: 4

Author: Eleanor Boyd (@eleanorjboyd)

Create Issue

Prerequisites

Use the following repo of tests or use your own repo of pytests https://github.com/eleanorjboyd/inc_dec_example_repo

Background Information

This rewrite is for the underlying infrastructure of how python handles testing therefore all behavior visible to the user should be pretty much the same (minus maybe some error logs going in a better place or test names shortened). Therefore throughout this testing, you can reference the current testing in Python to compare if you think something doesn't look right (or you can always post here too!).

Steps

Part 0: Setup

1. make sure you have the setting "python.experiments.optInto": ["All"]
2. Open the inc_dec_example_repo/inc_dec_example_test_suite testing repo folder in vscode
3. Use the create environment command from the command palette, select venv and whatever python version you want >=3.7
   in the terminal run python -m pip install pytest
4. From the command palette run python: configure tests select pytest and "." (root directory)

Part 1: UI Nesting & Run

1. Go to the testing explorer, and confirm all the tests are discovered

Part 2: Debugging

1. add some break points to the pytest tests and check to make sure these break points are hit
2. make sure you can do other normal debugging actions (stop, restart, step over, etc)

Part 3: Large Files

1. In test_parameterized.py, change @pytest.mark.parametrize("num", range(1, 100)) to be much larger like @pytest.mark.parametrize("num", range(1, 10000)) and make sure it both works for discovery and run tests (tests should be displayed on the test explorer after discovery and run).

Part 4: Seeing Dynamic Run

1. Expand the testing explorer UI to see all the parameterized tests under test_wait_parameterized.py so you can see the nodes 1 - 10.
2. Click run all and make sure that you are seeing the nodes on the test UI finish at different times (ie they don't all resolve together, like some are still spinning while others are complete with a check or an x)