Skip to content

README.md: better document test_stubtest pitfall #3793

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
srittau merged 1 commit into from
Feb 29, 2020
Merged

README.md: better document test_stubtest pitfall #3793

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,18 +162,23 @@ This test requires Python 3.5 or higher.
Run using `(.venv3)$ python3 tests/stubtest_test.py`

This test compares the stdlib stubs against the objects at runtime. Because of
this, the output depends on which version of Python it is run with.
this, the output depends on which version of Python and on what kind of system
it is run.
Thus the easiest way to run this test is by enabling Travis CI on your fork;
if you run it locally, it'll likely complain about system-specific
differences (in e.g, `socket`) that the type system cannot capture.
If you need a specific version of Python to repro a CI failure,
[pyenv](https://github.com/pyenv/pyenv) can help (as can enabling Travis CI on
your fork).
[pyenv](https://github.com/pyenv/pyenv) can help.

Due to its dynamic nature, you may run into false positives. In this case, you
can add to the whitelists for each affected Python version in
`tests/stubtest_whitelists`. Please file issues for stubtest false positives
at [mypy](https://github.com/python/mypy/issues).

To run stubtest against third party stubs, it's easiest to use stubtest
directly. stubtest can also help you find things missing from the stubs.
directly, with `(.venv3)$ python3 -m mypy.stubtest --custom-typeshed-dir
<path-to-typeshed> <third-party-module>`.
stubtest can also help you find things missing from the stubs.


### flake8
Expand Down
4 changes: 2 additions & 2 deletions tests/stubtest_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,8 @@ def run_stubtest(typeshed_dir: Path) -> int:
subprocess.run(cmd, check=True)
except subprocess.CalledProcessError as e:
print(
"\nNB: stubtest output depends on the Python version it is run with. See README.md "
"for more details.\n"
"\nNB: stubtest output depends on the Python version (and system) it is run with. "
"See README.md for more details.\n"
"NB: We only check positional-only arg accuracy for Python 3.8.\n"
"If stubtest is complaining about 'unused whitelist entry' after your fix, please "
"remove the entry from the whitelist file. Note you may have to do this for other "
Expand Down