Repository navigation
CI, TYP: Large-scale integration type-testing with mypy_primer #28054
Description
Our type-tests aren't perfect, we don't type-check our stubs, and don't use stubtest. But even if we would do all that, there's always a chance that seemingly innocent changes to the stubs could unintentionally lead to unforeseen typing issues in downstream packages.
The mypy_primer tool was built to help with issues like these, and helps reduce the number of these static-typing-related unknown unknowns.
From https://github.com/hauntsaninja/mypy_primer README.MD:
mypy_primer is a tool for regression testing Python static type checkers.
mypy_primer makes it easy to run different versions of a type checker over several million lines of open source Python projects with typing for the purpose of evaluating changes.
Here's what mypy_primer does:
- Clones a copy of mypy (potentially from a fork you specified)
- Checks out a "new" and "old" revision of mypy
- Clones a hardcoded list of projects (potentially filtered by you)
- Installs necessary stubs and dependencies per project
- Runs the appropriate mypy command per project
- Shows you the potentially differing results!
- Lets you bisect to find the mypy commit that causes a given change
I have little knowledge about NumPy's CI, but I think it could help a lot if we could incorporate mypy_primer in it, and preferably run it only if changes are made to at least one .pyi stub file.
Activity
jorenham commentedon Dec 22, 2024
python/typeshed#11830 (comment) and python/typeshed#13021 (comment) are examples of how it's used by
typeshedstefanv commentedon Dec 23, 2024
This would be very useful. It may also be helpful to do this on a cron against the nightly wheels. And we may need to choose mypy vs pyright, depending on what the project itself uses (they may no longer be making an effort to be compatible with mypy, e.g.).
jorenham commentedon Dec 23, 2024
There will be cases when we will deliberately choose to ignore the mypy_primer output. So I can imagine that doing this periodically would be more annoying than it would be helpful 🤔.
Unlike the name suggests,
mypy_primerisn't exclusive tomypy. If I understand correctly, for each project it can be configured to usemypy,pyright, both, or forks (e.g.basedmypyorbasedpyright).rgommers commentedon Dec 24, 2024
Isn't the problem that there are very few external projects to run CI on that actually have good type annotations? I'm thinking that a single job to sometimes run against
scipy-stubs(anything else?) that you trust is good, but much more is going to be a pain.The usual solution for this problem that we encourage is for downstream packages to test numpy nightlies. Quite a few projects do this now, so getting them to run their typing tests as well on those nightlies if they have any is probably more effective.
stefanv commentedon Dec 24, 2024
Yes, we'd be happy to add a daily/weekly check like that to skimage and the SP Lecture Notes, e.g.
jorenham commentedon Dec 24, 2024
It doesn't matter if the projects are currently correctly typed, or even typed at all. Mypy_primer reports the diffs of the mypy/pyright outputs of that project, that our change causes.
hauntsaninja commentedon Dec 28, 2024
Awesome that you already thought of this! I went ahead and added a few features to mypy_primer to make this easy to add to numpy.
--known-dependency-selector numpy. This lets you select all projects in mypy_primer's corpus where we at some point recorded that numpy is a dependency. There are currently 19 of these.--(old/new)-prepend-path. This lets you effectively add a new entry tosys.pathfor the new and old runs. Two gotchas: a) you'll want to make sure there's anumpydir and anumpy/py.typedin the path you prepend, b) there may be some weirdness with namespace packages (which I guess doesn't apply to numpy)I was able to get some potentially believable output with:
where I checked out numpy v2.1.3 and v2.2.0 to nump1 and nump2 respectively. Make sure to explicitly set
--newand--oldto the same thing, otherwise you'll also get type checker version differences.mypy_primer supports pyright and mypy. It probably also supports basedpyright via the
--repoflag.I do see mypy hit some errors when using v2.1.3 on the lines of
AssertionError: numpy._typing._nbit_base._8Bit. This is somewhat surprising, because we should have seen those in mypy and typeshed CI. Any chance that error is at all familiar to you? It's possible I messed something up.Anything I should know before opening a PR?
jorenham commentedon Dec 28, 2024
Wow, thanks a lot for that @hauntsaninja, that indeed sounds very helpful.
Hmm, maybe it has something to do with the mypy plugin (https://github.com/numpy/numpy/blob/main/numpy/typing/mypy_plugin.py)? It is used for system-dependent scalar types, like
np.longdouble, so that the types match match the runtime, for which it uses those_typing._nbit_basetypes. But I don't see any asserts in the mypy plugin itself, so theAssertionErrorcould then perhaps come from mypy itself or something?🤷🏻
Nothing in particular I suppose.
Just don't forget to use a
CI:prefix for your commit messages and PR title (or whichever one seems appropriate, see https://numpy.org/doc/stable/dev/development_workflow.html#writing-the-commit-message).TYP: use mypy_primer to surface type checking regressions
TYP: use mypy_primer to surface type checking regressions
hauntsaninja commentedon Dec 28, 2024
Thanks! Ah, the plugin is a good guess. It looks like the plugin imports numpy but I didn't bother rebuilding numpy, so it's possible I introduced some skew.