Repository navigation
regrtest: add --list-cases option to only display test case identifiers #74708
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2017-06-26.09:47:46.672>
created_at = <Date 2017-05-31.09:22:37.791>
labels = ['3.7', 'type-feature', 'tests']
title = 'regrtest: add --list-cases option to only display test case identifiers'
updated_at = <Date 2017-06-27.14:56:45.897>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2017-06-27.14:56:45.897>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-06-26.09:47:46.672>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2017-05-31.09:22:37.791>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 30523
keywords = []
message_count = 26.0
messages = ['294825', '294828', '294829', '294830', '294840', '295084', '295087', '295146', '295166', '295500', '296158', '296170', '296175', '296176', '296177', '296180', '296189', '296201', '296207', '296208', '296863', '296864', '296884', '297035', '297042', '297045']
nosy_count = 7.0
nosy_names = ['vstinner', 'rbcollins', 'ezio.melotti', 'michael.foord', 'serhiy.storchaka', 'mdk', 'louielu']
pr_nums = ['1885', '2238', '2243', '2244', '2249', '2250', '2401', '2441', '2442', '2444']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue30523'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']
Activity
vstinner commentedon May 31, 2017
Attached pull requests adds a --list-tests option to only display the list of test names, don't run tests.
It's a first step to implement the issue bpo-29512: "regrtest refleak: implement bisection feature".
vstinner commentedon May 31, 2017
Maybe the feature already exists but I failed to find it. Since I'm suprised that nobody implements that before (in unittest), I wait for a first feedback before starting to document the new feature and to write an unit test for it.
support.run_unittest() doesn't use unittest.main() and so don't support the new --list-tests option.
vstinner commentedon May 31, 2017
I wrote this patch because I'm trying to find which test of test_threading leaks a reference:
./python -m test -R 3:3 test_threading
Usually, I modify Lib/test/test_threading.py to remove tests one by one. I "bisect" the test manually. But this process is slow and painful, and it seems like we have many tests which leak references!
JulienPalard commentedon May 31, 2017
If it can be of any help, pytest have a "pytest --collect-only", it may make sense to use the same option name for consistency. I also though about "--dry-run" but I still prefer --list-tests or --collect-only.
vstinner commentedon May 31, 2017
I used testr which has a --list-tests option:
http://testrepository.readthedocs.io/en/latest/MANUAL.html#listing-tests
serhiy-storchaka commentedon Jun 3, 2017
How to use this feature? "./python -m unittest --list-tests" produces nothing.
JulienPalard commentedon Jun 3, 2017
It works in "discover" mode, I tested:
./python -m unittest discover ./Lib/unittest/test/ --list-tests
and it worked.
So maybe the --list-tests should be moved to _getDiscoveryArgParser?
Also spotted that tests are printed on stderr, typically when the users asks for something, it has to be printed to stdout. Typically usages are written on stderr in case of error, but on stdout when explicitly asked via --help, see:
vstinner commentedon Jun 4, 2017
I used ./python -m test Lib/test/test_threading.py --list-tests.
I would like to be able to use this feature in regrtest too:
./python -m test test_threading --list-tests
Sadly, this option already exists and only return test files, not test
methods.
serhiy-storchaka commentedon Jun 5, 2017
I concur with Julien, the result should be printed to stdout.
I tried:
All these doctest.DocTestCase.runTest look useless.
But "./python -m unittest test.test_json --list-tests" and "./python -m test.test_json --list-tests" give the same output.
"./python -m test.test_doctest --list-tests" and "./python -m test.test_tk --list-tests" run tests instead of listing them.
serhiy-storchaka commentedon Jun 9, 2017
The main problem is doctests. Numerous "doctest.DocTestCase.runTest" look useless and confusing.
Running tests instead of listing them is bad.
It is not clear why in one cases --list-tests outputs full qualified test names, while in other cases it outputs only class and method names.
mlouielu commentedon Jun 16, 2017
Serhiy and Victor suggest moving this option to regrtest. open a new PR 2238 addressed this.
Also, because moving into regrtest, some test that used
requireswill need to pass option when using it. e.g.:$ ./python -m test test.test_curses --list-cases -ucursesotherwise, it will just passout and do nothing (maybe it should warn user?)
vstinner commentedon Jun 16, 2017
New changeset a49c935 by Victor Stinner (mlouielu) in branch 'master':
bpo-30523: regrtest: Add --list-cases option (bpo-2238)
a49c935
2 remaining items
vstinner commentedon Jun 16, 2017
New changeset b0c58d3 by Victor Stinner (Louie Lu) in branch 'master':
bpo-30523: Add --list-cases unittest (bpo-2243)
b0c58d3
vstinner commentedon Jun 16, 2017
I created bpo-30683: Enhance doctest support in regrtest --list-cases.
vstinner commentedon Jun 16, 2017
New changeset a0ccc54 by Victor Stinner in branch '3.6':
Synchronize libregrtest from master to 3.6 (bpo-2244)
a0ccc54
vstinner commentedon Jun 16, 2017
New changeset 24c2c20 by Victor Stinner in branch '2.7':
bpo-30540, bpo-30523: Add --matchfile and --list-cases options to regrtest (bpo-2249)
24c2c20
[-]unittest: add --list-tests option to only display the list of test names, don't run tests[/-][+]regrtest: add --list-cases option to only display test case identifiers[/+]vstinner commentedon Jun 16, 2017
On Python 2.7, listing *all* test cases with "./python -m test --list-cases" fails on the following test files:
regrtest of Python 2.7 uses the test_main() function of each test file, if the function is defined. Whereas --list-cases is more written as a raw discovery function listing all cases which inherit from unittest.TestCase. These 3 files have base test cases which are "abstract" or configured below.
Options:
I would prefer to limit changes in Python 2.7 branch, so IMHO the second option is the best: modify the 3 tests.
vstinner commentedon Jun 16, 2017
New changeset 46e299c by Victor Stinner in branch '3.5':
[3.5] bpo-30540, bpo-30523: Add --matchfile and --list-cases options to regrtest (bpo-2250)
46e299c
vstinner commentedon Jun 26, 2017
I created bpo-30759: "[2.7] Fix python2 -m test --list-cases test_multibytecodec_support test_robotparser".
vstinner commentedon Jun 26, 2017
regrtest now supports --list-cases and --matchfile features in 2.7, 3.5, 3.6 and master (future 3.7) branches, so I now close the issue.
See bpo-30759 for the pending bug in Python 2.7. IMHO this bug is not a blocking feature, so it's ok to this close this issue.
Thanks Serhiy Storchaka and Julien Palard for the feedback and reviews, thanks Louie Lu for the actual implementation, much simpler than what I expected ;-)
These features already helped me a lot of identified reference leaks. So thank you all ;-)
vstinner commentedon Jun 26, 2017
New changeset ace56d5 by Victor Stinner in branch 'master':
bpo-30523: regrtest --list-cases --match (bpo-2401)
ace56d5
vstinner commentedon Jun 27, 2017
New changeset 35d2ca2 by Victor Stinner in branch '3.6':
[3.6] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (bpo-2441)
35d2ca2
vstinner commentedon Jun 27, 2017
New changeset de1850b by Victor Stinner in branch '3.5':
[3.5] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (bpo-2442)
de1850b
vstinner commentedon Jun 27, 2017
New changeset fea98bf by Victor Stinner in branch '2.7':
[2.7] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (bpo-2444)
fea98bf