You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
mypy 0.990 is 1000x slow on pydantic codebase vs 0.982 #14034
As with #14027, would it be worth adding pydantic to mypy_primer @hauntsaninja?
As with sqlalachemy, pydantic pushes python's type system fairly hard, in particular, the schema for pydantic-core, is defined as a recursive union of TypedDicts.
OK, then this will be not easy to fix. The problem is that you have that big union of 40 items. Previously mypy didn't support recursive types, so most likely all recursive types were either plain Any or shallow TypedDicts with Any. Mypy used to give an error about this however. I guess this error was not shown because pydantic_core is checked as third party dependency? You can try --no-silence-site-packages when using --disable-recursive-aliases. (Btw everything on PYTHONPATH is considered third party, cc @JukkaL)
So now when recursive types are properly supported imagine you have a triple nested dictionary typed as CoreSchema. Mypy will need to try every item in the union until it finds a matching one. This will take on average 20 * 20 * 20 = 8000 type checks for a single dict. To make things worse, TypedDicts are structural, and some have overlapping keys so it may take a while to figure out that a given item doesn't actually match.
We will still try to profile this. Maybe there is some room for optimizations.
Simple answer might be to move to pyright, which we already use to check pydantic-core and seems to do a reasonably good job (although that union found a bug in pyright too). (I don't mean this to sound hostile, mypy is amazing and I'd love to continue to use it, but I understand if this is a fundamental limitation of the library)
If it helps, all those TypedDicts have a type key which can (mostly) be used as a discriminator - that's how we validate pydantic-core schema using pydantic-core, see here. Using that discriminator, the number of types checked could be massively reduced, but I guess that's too specific for mypy to be able to pick up on.
Btw if you fix the errors coming from no_implicit_optional=True the cache should became effective again. I guess even with recursive types enabled, a typical run should take just few seconds with warm cache.
Thanks, pydantic is already in mypy_primer :-) However, we don't currently monitor perf regressions in mypy_primer... it's a feature I've been thinking about though
I did some testing and came up with a self-contained example that may be helpful.
It seems that both the size of the union and how many times SchemaXX() gets called both matter. Using non-overlapping keys (e.g. type_99) doesn't seem to matter. Removing the recursive references does indeed improve things.
from pydantic import BaseModel
class MyModel(BaseModel):
x: int
Running mypy on this file takes about 16s for me with no cache (with both mypy versions 1.1.1 and 1.2 both compiled, pydantic version 1.10.7, no extra mypy configuration)
Obviously it would be preferable to actually solve the issue, but note that one potential hack is to use a MYPY constant. mypy will always treat this as True, so you could do something like:
MYPY = False
if MYPY:
BigSchemaType = Any # or possibly a slightly better fallback
else:
BigSchemaType = Union[...]
Running mypy on this file takes about 16s for me with no cache (with both mypy versions 1.1.1 and 1.2 both compiled, pydantic version 1.10.7, no extra mypy configuration)
I believe the issue is only present in the current dev builds for pydantic as a result of the upcoming move to version 2. I can reproduce the issue with pydantic==2.0a1 and --no-incremental.
To reproduce
Create a new venv (I'm using Python 3.11)
pip install -U pip wheel setuptools (might not be necessary)
pip install mypy==1.2.0 pydantic==2.0a1
mypy -v --no-incremental example.py
# example.pyfrompydanticimportBaseModelclassMyModel(BaseModel):
x: int
Mypy starts to hang at
LOG: Processing SCC singleton (pydantic_core.core_schema) as inherently stale with stale deps (__future__ builtins datetime sys typing)
Correct @cdce8p, my mistake - I was running on main but the same will be true on 2.0a1 - or indeed any V2 release until we use a hack like if not MYPY: on this line, or mypy fixes some of these performance issues.
~/test λ pip freeze
annotated-types==0.4.0
mypy==1.2.0
mypy-extensions==1.0.0
pydantic==2.0a1
pydantic_core==0.23.0
tomli==2.0.1
typing_extensions==4.5.0
~/test λ mypy --version
mypy 1.2.0 (compiled: yes)
λ cat example.py
from pydantic import BaseModel
class MyModel(BaseModel):
x: int
~/test λ time mypy --no-incremental example.py
Success: no issues found in 1 source file
mypy --no-incremental example.py 2.96s user 0.16s system 99% cpu 3.124 total
Yes, we basically used that constant to the disable type checking for mypy and will have to tell our users that if they want proper type checkin they need to use a different type checker 😔
Activity
samuelcolvin commentedon Nov 8, 2022
As with #14027, would it be worth adding pydantic to mypy_primer @hauntsaninja?
As with sqlalachemy, pydantic pushes python's type system fairly hard, in particular, the schema for pydantic-core, is defined as a recursive union of TypedDicts.
ilevkivskyi commentedon Nov 8, 2022
Note that when an error is present in a file, mypy will not use cache for that file.
ilevkivskyi commentedon Nov 8, 2022
Btw can you try
--disable-recursive-aliases? Such big recursive types are known to be slow.samuelcolvin commentedon Nov 8, 2022
Yup, that makes a big different, time reduces to 9s.
ilevkivskyi commentedon Nov 8, 2022
Also, just to be sure, can you also try
--enable-recursive-aliaseson 0.982?samuelcolvin commentedon Nov 8, 2022
Takes 261s as you predicted - seems that's the problem.
ilevkivskyi commentedon Nov 8, 2022
OK, then this will be not easy to fix. The problem is that you have that big union of 40 items. Previously mypy didn't support recursive types, so most likely all recursive types were either plain
Anyor shallow TypedDicts withAny. Mypy used to give an error about this however. I guess this error was not shown becausepydantic_coreis checked as third party dependency? You can try--no-silence-site-packageswhen using--disable-recursive-aliases. (Btw everything onPYTHONPATHis considered third party, cc @JukkaL)So now when recursive types are properly supported imagine you have a triple nested dictionary typed as
CoreSchema. Mypy will need to try every item in the union until it finds a matching one. This will take on average 20 * 20 * 20 = 8000 type checks for a single dict. To make things worse, TypedDicts are structural, and some have overlapping keys so it may take a while to figure out that a given item doesn't actually match.We will still try to profile this. Maybe there is some room for optimizations.
samuelcolvin commentedon Nov 8, 2022
Thanks so much for the explanation.
Simple answer might be to move to pyright, which we already use to check pydantic-core and seems to do a reasonably good job (although that union found a bug in pyright too). (I don't mean this to sound hostile, mypy is amazing and I'd love to continue to use it, but I understand if this is a fundamental limitation of the library)
If it helps, all those TypedDicts have a
typekey which can (mostly) be used as a discriminator - that's how we validate pydantic-core schema using pydantic-core, see here. Using that discriminator, the number of types checked could be massively reduced, but I guess that's too specific for mypy to be able to pick up on.ilevkivskyi commentedon Nov 8, 2022
Btw if you fix the errors coming from
no_implicit_optional=Truethe cache should became effective again. I guess even with recursive types enabled, a typical run should take just few seconds with warm cache.samuelcolvin commentedon Nov 8, 2022
ye, I thought that, but it's still not really acceptable for either CI or new contributors for mypy to take minutes.
I guess we'll have to use
--disable-recursive-aliases.hauntsaninja commentedon Nov 8, 2022
Thanks, pydantic is already in mypy_primer :-) However, we don't currently monitor perf regressions in mypy_primer... it's a feature I've been thinking about though
adriangb commentedon Mar 15, 2023
I did some testing and came up with a self-contained example that may be helpful.
It seems that both the size of the union and how many times
SchemaXX()gets called both matter. Using non-overlapping keys (e.g.type_99) doesn't seem to matter. Removing the recursive references does indeed improve things.Long code block
2 remaining items
hauntsaninja commentedon Apr 7, 2023
Hmm...
Running mypy on this file takes about 16s for me with no cache (with both mypy versions 1.1.1 and 1.2 both compiled, pydantic version 1.10.7, no extra mypy configuration)
Obviously it would be preferable to actually solve the issue, but note that one potential hack is to use a
MYPYconstant. mypy will always treat this as True, so you could do something like:adriangb commentedon Apr 7, 2023
The issue is that if you use those types that 16s grows exponentially (give or take).
We are indeed exploring that: pydantic/pydantic-core#528. Is this approach documented anywhere?
hauntsaninja commentedon Apr 7, 2023
Yeah, makes sense, sounds like Samuel didn't paste the full example.py he was running against in #14034 (comment) then
MYPY constant is documented, but not particularly thoroughly, e.g. at https://mypy.readthedocs.io/en/stable/common_issues.html#python-version-and-system-platform-checks Too much of its use would probably be bad for the typing ecosystem
cdce8p commentedon Apr 7, 2023
I believe the issue is only present in the current dev builds for pydantic as a result of the upcoming move to version 2. I can reproduce the issue with
pydantic==2.0a1and--no-incremental.To reproduce
pip install -U pip wheel setuptools(might not be necessary)pip install mypy==1.2.0 pydantic==2.0a1mypy -v --no-incremental example.pyMypy starts to hang at
samuelcolvin commentedon Apr 7, 2023
Correct @cdce8p, my mistake - I was running on
mainbut the same will be true on2.0a1- or indeed any V2 release until we use a hack likeif not MYPY:on this line, or mypy fixes some of these performance issues.hauntsaninja commentedon Apr 7, 2023
@cdce8p weird, I still can't repro
hauntsaninja commentedon Apr 7, 2023
Oh looks like pydantic_core made a new release (presumably including the if MYPY hack)
adriangb commentedon Apr 7, 2023
Yes, we basically used that constant to the disable type checking for mypy and will have to tell our users that if they want proper type checkin they need to use a different type checker 😔
hauntsaninja commentedon Apr 21, 2023
Okay, took a look tonight and found a simple patch that makes this terminate (but is still extremely slow).
Note for self:
pip install 'pydantic-core==0.22.0' 'pydantic==2.0a1'to get the right versions.Fix performance in union subtyping
Fix performance in union subtyping (#15104)
JukkaL commentedon Apr 24, 2023
#14150 further improved performance of the simple example by 3.5x for me (when not using a compiled mypy).
Fix performance in union subtyping (#15104)
hauntsaninja commentedon Apr 26, 2023
#15128 should be another 25% for me on the pydantic codebase