Unify ways to run the Python test suite #109566

There are at least 15 known ways to run the Python test suite. IMO it's too much, it's hard to keep all these code paths consistent. IMO python -m test should be enough to fit all use cases.

Maybe we need a --ci, --buildbot, or --strict option which would run tests is "strict" mode.

Portable way to run tests:

./python -m test
./python -m test.regrtest
./python -m test.libregrtest
./python -m test.autotest

./python Lib/test/regrtest.py
./python Lib/test/autotest.py

./python -c 'from test import autotest'

The main drawback is that running ./python doesn't work when Python is built with ./configure --enable-shared, and it doesn't work when Python is cross-compiled (WASM/WASI buildbots). Moreover, on macOS, ./python.exe must be run instead. On Windows, it's just python.

Unix, from Python source code:

make buildbottest TESTOPTS="..."
make test
make testall
make testuniversal
make hostrunnertest

./python Tools/scripts/run_tests.py

Windows, from Python source code:

PCbuild\rt.bat
Tools\buildbot\test.bat

• Lib/test/autotest.py just runs test.libregrtest.main().

• Lib/test/regrtest.py runs test.libregrtest.main() and remove Lib/test/ from sys.path and make __file__ an absolute path.

• PCbuild\rt.bat:

  • python: pass -u -Wd -E -bb options
  • regrtest: pass options passed to rt.bat (except of options specific to rt.bat)
  • most options are to get the path to the python.exe program in PCbuild/

• Tools\buildbot\test.bat runs PCbuild\rt.bat:

  • rt.bat: pass -q -d options.
  • regrtest: pass -j1 -uall -rwW --slowest --timeout=1200 --fail-env-changed options.
  • ARM32 SSH pass -unetwork -udecimal -usubprocess -uurlfetch -utzdata -rwW --slowest --timeout=1200 --fail-env-changed options.

• make buildbottest runs Tools/scripts/run_tests.py:

  • python: pass $(TESTPYTHONOPTS) options.
  • regrtest: pass -j 1 -u all -W --slowest --fail-env-changed --timeout=$(TESTTIMEOUT) $(TESTOPTS) options.

• Tools/scripts/run_tests.py:

  • python: pass -u -W default -bb -E options -- also cross-compilation options, not detailed here
  • regrtest: pass -r -w -j0 -u all,-largefile,-audio,-gui options, add -n on Windows.

GitHub Action workflow uses, .github/workflows/build.yml:

• Windows x86: .\PCbuild\rt.bat -p Win32 -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0
• Windows x64: .\PCbuild\rt.bat -p x64 -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0
• macOS: make buildbottest TESTOPTS="-j4 -uall,-cpu"
• Ubuntu: xvfb-run make buildbottest TESTOPTS="-j4 -uall,-cpu"
• Address Sanitizer: xvfb-run make buildbottest TESTOPTS="-j4 -uall,-cpu"

Buildbot use, master/custom/factories.py:

• UnixBuild and UnixCrossBuild: make buildbottest TESTOPTS="..." TESTPYTHONOPTS="..." TESTTIMEOUT="..." with code to select TESTOPTS, TESTPYTHONOPTS and TESTTIMEOUT.
• UnixInstalledBuild: /path/to/python -m test ... with code to select -m test options.
• BaseWindowsBuild: Tools\buildbot\test.bat ... with code to select .... options

Linked PRs

• gh-109566: Run buildbot tests with --fail-rerun #109567
• gh-109566, regrtest: Add --fast-ci and --slow-ci options #109570
• gh-109566: Fix typo in PCbuild/rt.bat #109701
• gh-109566: regrtest reexecutes the process #109909
• gh-109566: Fix regrtest code adding Python options #109926
• gh-109566: Fix regrtest Python options for WASM/WASI #109954
• gh-109566: regrtest doesn't enable --rerun if --python is used #109969
• gh-109566: regrtest _add_python_opts() handles KeyboardInterrupt #110062
• gh-109566: PCbuild/rt.bat now uses --fast-ci #110120
• gh-109566: regrtest --fast-ci no longer enables --nowindow #110121
• gh-109566: Remove make testall #110122