Narrow type variable bounds in binder by ilevkivskyi · Pull Request #19183 · python/mypy

ilevkivskyi · 2025-05-31T14:02:45Z

Fixes #5720
Fixes #8556
Fixes #9778
Fixes #10003
Fixes #10817
Fixes #11163
Fixes #11664
Fixes #12882
Fixes #13426
Fixes #13462
Fixes #14941
Fixes #15151
Fixes #19166

This handles a (surprisingly) common edge case. The charges in bind_self() and bind_self_fast() are tricky. I got a few "Redundant cast" errors there, which seemed good, but then I realized that attribute access etc. on a type variable go through slow PyObject paths, so I am actually forcing CallableType instead of F(bound=CallableType) there, since these are performance-critical functions.

ilevkivskyi · 2025-05-31T21:25:35Z

OK, the mypy_primer results are good, and although it looks like a niche use case, this PR fixes more than dozen issues (some with many upvotes).

hauntsaninja · 2025-06-01T00:27:57Z

Hooray!

…unds

github-actions · 2025-06-03T18:36:39Z

Diff from mypy_primer, showing the effect of this PR on open source code:

PyGithub (https://github.com/PyGithub/PyGithub)
+ github/MainClass.py:988: error: Unused "type: ignore" comment  [unused-ignore]

antidote (https://github.com/Finistere/antidote): 1.47x slower (65.5s -> 96.3s in single noisy sample)

prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/states.py:369: error: Argument 1 to "StateGroup" has incompatible type "Collection[Union[State[Any], Any]]"; expected "list[State[Any]]"  [arg-type]
+ src/prefect/states.py:369: error: Argument 1 to "StateGroup" has incompatible type "Collection[Never]"; expected "list[State[Any]]"  [arg-type]
+ src/prefect/futures.py:222: note:     def result(self, raise_on_failure: Literal[True] = ..., retry_result_failure: bool = ...) -> Any
+ src/prefect/futures.py:222: note:     def result(self, raise_on_failure: Literal[False] = ..., retry_result_failure: bool = ...) -> Union[Any, Exception]
+ src/prefect/futures.py:222: note:     def result(self, raise_on_failure: bool = ..., retry_result_failure: bool = ...) -> Union[Any, Exception]
- src/prefect/task_engine.py:467: error: Item "type" of "Union[ResultRecord[Any], type[NotSet]]" has no attribute "result"  [union-attr]
+ src/prefect/task_engine.py:467: error: Item "type" of "Union[R, type[NotSet]]" has no attribute "result"  [union-attr]
- src/prefect/task_engine.py:1022: error: Item "type" of "Union[ResultRecord[Any], type[NotSet]]" has no attribute "result"  [union-attr]
+ src/prefect/task_engine.py:1022: error: Item "type" of "Union[R, type[NotSet]]" has no attribute "result"  [union-attr]
- src/prefect/flow_runs.py:292: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "load"  [union-attr]
+ src/prefect/flow_runs.py:292: error: "type[T]" has no attribute "load"  [attr-defined]
- src/prefect/flow_runs.py:307: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "save"  [union-attr]
+ src/prefect/flow_runs.py:307: error: "type[T]" has no attribute "save"  [attr-defined]
- src/prefect/flow_runs.py:319: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "load"  [union-attr]
+ src/prefect/flow_runs.py:319: error: "type[T]" has no attribute "load"  [attr-defined]
- src/prefect/flow_runs.py:446: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "load"  [union-attr]
+ src/prefect/flow_runs.py:446: error: "type[T]" has no attribute "load"  [attr-defined]
- src/prefect/flow_runs.py:456: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "save"  [union-attr]
+ src/prefect/flow_runs.py:456: error: "type[T]" has no attribute "save"  [attr-defined]

kopf (https://github.com/nolar/kopf)
+ kopf/_cogs/structs/dicts.py:243: error: Unused "type: ignore" comment  [unused-ignore]
+ kopf/_core/intents/handlers.py:36: error: Redundant cast to "execution.CauseT"  [redundant-cast]

python-chess (https://github.com/niklasf/python-chess)
- chess/pgn.py:1315: error: Returning Any from function declared to return "HeadersT"  [no-any-return]

kornia (https://github.com/kornia/kornia)
+ kornia/utils/helpers.py:387: error: Unused "type: ignore" comment  [unused-ignore]

mypy (https://github.com/python/mypy)
+ mypy/expandtype.py:125: error: Redundant cast to "F"  [redundant-cast]
+ mypy/expandtype.py:125: note: See https://mypy.rtfd.io/en/stable/_refs.html#code-redundant-cast for more info
+ mypy/typeops.py:421: error: Redundant cast to "F"  [redundant-cast]
+ mypy/typeops.py:430: error: Redundant cast to "F"  [redundant-cast]
+ mypy/typeops.py:469: error: Incompatible types in assignment (expression has type "CallableType", variable has type "F")  [assignment]
+ mypy/typeops.py:484: error: Redundant cast to "F"  [redundant-cast]
+ mypy/checkmember.py:1489: error: Redundant cast to "F"  [redundant-cast]
+ mypy/checkmember.py:1492: error: Redundant cast to "F"  [redundant-cast]
+ mypy/checkmember.py:1502: error: Redundant cast to "F"  [redundant-cast]

hydra-zen (https://github.com/mit-ll-responsible-ai/hydra-zen)
- src/hydra_zen/third_party/beartype.py:128: error: Incompatible types in assignment (expression has type "_T", variable has type "type[Any]")  [assignment]
+ src/hydra_zen/third_party/beartype.py:128: error: Incompatible types in assignment (expression has type "_T", variable has type "_T")  [assignment]
+ src/hydra_zen/third_party/beartype.py:130: error: Redundant cast to "_T"  [redundant-cast]
+ src/hydra_zen/third_party/pydantic.py:111: error: Redundant cast to "_T"  [redundant-cast]
+ src/hydra_zen/third_party/pydantic.py:115: error: Redundant cast to "_T"  [redundant-cast]
- src/hydra_zen/third_party/pydantic.py:123: error: Argument 1 has incompatible type "_T | type[object]"; expected "_T"  [arg-type]

pytest (https://github.com/pytest-dev/pytest)
- src/_pytest/fixtures.py:1132: error: Returning Any from function declared to return "FixtureValue"  [no-any-return]

pandas-stubs (https://github.com/pandas-dev/pandas-stubs)
+ tests/__init__.py:67: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/__init__.py:85: error: Unused "type: ignore" comment  [unused-ignore]

beartype (https://github.com/beartype/beartype)
+ beartype/_decor/_type/decortype.py:133: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_type/decortype.py:295: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_nontype/decornontype.py:294: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_nontype/decornontype.py:301: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_nontype/decornontype.py:380: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_nontype/decornontype.py:395: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_nontype/decornontype.py:567: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/decorcore.py:133: error: Unused "type: ignore" comment  [unused-ignore]

xarray (https://github.com/pydata/xarray)
+ xarray/structure/alignment.py: note: In function "_broadcast_helper":
+ xarray/structure/alignment.py:1140: error: Redundant cast to "T_Alignable"  [redundant-cast]
+ xarray/structure/alignment.py:1142: error: Redundant cast to "T_Alignable"  [redundant-cast]

steam.py (https://github.com/Gobot1234/steam.py)
- steam/ext/commands/commands.py:284: error: Argument 1 to "append" of "list" has incompatible type "Command[Any, Any, Any]"; expected "Self"  [arg-type]
- steam/ext/commands/cog.py:182: error: Argument 1 to "decorator" has incompatible type "Callable[[Any | None, VarArg(Any), KwArg(Any)], Coroutine[Any, Any, None]]"; expected "F"  [arg-type]
- steam/ext/commands/bot.py:415: error: Argument 1 to "decorator" has incompatible type "Callable[..., Coroutine[Any, Any, Any]]"; expected "F"  [arg-type]

netomi · 2025-06-11T19:16:11Z

ftr: this commit fixes a bug we encountered with mypy 1.16.0 (and not in 1.15.0) in different places with a stacktrace like:

asfyaml/feature/github/pull_requests.py: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 1.16.0
Traceback (most recent call last):
  File "/home/tn/workspace/apache/infrastructure-asfyaml/.venv/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "mypy/build.py", line 2062, in wrap_context
RuntimeError: ('Not implemented', <class 'mypy.types.TypeGuardedType'>)
asfyaml/feature/github/pull_requests.py: : note: use --pdb to drop into pdb

ilevkivskyi · 2025-06-11T19:57:13Z

@netomi Hm, this is interesting, could you please post a full traceback? (use --show-traceback)

netomi · 2025-06-11T20:14:27Z

that is already with this option:

tn@proteus:~/workspace/apache/infrastructure-asfyaml$ poetry run mypy --show-traceback asfyaml/feature/github/pull_requests.py 
asfyaml/feature/github/pull_requests.py: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 1.16.0
Traceback (most recent call last):
  File "/home/tn/workspace/apache/infrastructure-asfyaml/.venv/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "mypy/build.py", line 2062, in wrap_context
RuntimeError: ('Not implemented', <class 'mypy.types.TypeGuardedType'>)
asfyaml/feature/github/pull_requests.py: : note: use --pdb to drop into pdb

the project can be found here: https://github.com/apache/infrastructure-asfyaml

it is now using 1.15.0, but updating to 1.16.0 shows the bug.

ilevkivskyi · 2025-06-14T00:31:23Z

A bit more of the traceback

  File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/build.py", line 2062, in wrap_context
    yield
  File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/build.py", line 2391, in finish_passes
    self._patch_indirect_dependencies(self.type_checker().module_refs, all_types)
  File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/build.py", line 2424, in _patch_indirect_dependencies
    encountered = self.manager.indirection_detector.find_modules(types) | module_refs
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/indirection.py", line 35, in find_modules
    self._visit(typs)
  File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/indirection.py", line 46, in _visit
    typ.accept(self)
  File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/types.py", line 276, in accept
    raise RuntimeError("Not implemented", type(self))
RuntimeError: ('Not implemented', <class 'mypy.types.TypeGuardedType'>)

This looks like something where I would say this PR may be indeed a proper fix. So looks like no action needed (apart from releasing 1.17 LOL)

netomi · 2025-06-14T04:29:12Z

just wanted to mention it as there might be others that report issues about that, so there should be a changelog entry to track that this is solved in 1.17.0 imho.

Narrow type variable bounds in binder

fd8d1cd

This comment has been minimized.

Sign in to view

Reorder branches in narrowing logic

e2fadec

This comment has been minimized.

Sign in to view

Use a bit more principled handling of unions

fcc9af6

This comment has been minimized.

Sign in to view

This was referenced May 31, 2025

Checking Generic Type at Runtime #13053

Closed

Narrowing doesn't work on a typevar with a union bound #15631

Closed

Narrow TypeVars in the negative case + narrow them through upper bounds #15711

Closed

Narrow TypeVar to bounded TypeVar #15377

Closed

ilevkivskyi requested review from JukkaL and hauntsaninja May 31, 2025 21:25

hauntsaninja approved these changes Jun 1, 2025

View reviewed changes

sterliakov approved these changes Jun 1, 2025

View reviewed changes

sterliakov mentioned this pull request Jun 3, 2025

Problems with narrowing generic T | Sequence[T] #19202

Closed

ilevkivskyi mentioned this pull request Jun 3, 2025

1.16 Release Planning #18739

Closed

ilevkivskyi added 2 commits June 3, 2025 18:29

Merge remote-tracking branch 'upstream/master' into narrow-typevar-bo…

21a2153

…unds

Address offline feedback

54a80bb

ilevkivskyi merged commit 5a0fa55 into python:master Jun 3, 2025

ilevkivskyi deleted the narrow-typevar-bounds branch June 3, 2025 18:37

This was referenced Jun 5, 2025

mypy too strict in enforcing generic return types when they are unions #19239

Closed

An instance of a final class should be treated as satisfying a TypeVar bound to it, after being narrowed #19255

Open

sterliakov mentioned this pull request Jul 10, 2025

RuntimeError: Not implemented mypy.types.TypeGuardedType #19412

Closed

gopidesupavan mentioned this pull request Jul 10, 2025

Make task-sdk compatible with mypy 1.16.1 apache/airflow#53047

Merged

sterliakov mentioned this pull request Jul 16, 2025

Comparison operators emit "Unsupported operand types" when using isinstance to narrow TypeVar to union of numeric types #19454

Open

pkit mentioned this pull request Aug 16, 2025

Wrong type narrowing when using type without an argument in isinstance call #13462

Closed

hauntsaninja mentioned this pull request Feb 10, 2026

issubclass doesn't narrow down types #10978

Open

Navigation Menu

Repository navigation

Narrow type variable bounds in binder - #19183

Narrow type variable bounds in binder#19183
ilevkivskyi merged 5 commits into
python:masterfrom
ilevkivskyi:narrow-typevar-bounds

ilevkivskyi commented May 31, 2025 •

edited

Loading

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ilevkivskyi commented May 31, 2025

Uh oh!

hauntsaninja commented Jun 1, 2025

Uh oh!

github-actions Bot commented Jun 3, 2025

Uh oh!

netomi commented Jun 11, 2025 •

edited

Loading

Uh oh!

ilevkivskyi commented Jun 11, 2025

Uh oh!

netomi commented Jun 11, 2025 •

edited

Loading

Uh oh!

ilevkivskyi commented Jun 14, 2025

Uh oh!

netomi commented Jun 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ilevkivskyi commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ilevkivskyi commented May 31, 2025

Uh oh!

hauntsaninja commented Jun 1, 2025

Uh oh!

github-actions Bot commented Jun 3, 2025

Uh oh!

netomi commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilevkivskyi commented Jun 11, 2025

Uh oh!

netomi commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilevkivskyi commented Jun 14, 2025

Uh oh!

netomi commented Jun 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ilevkivskyi commented May 31, 2025 •

edited

Loading

netomi commented Jun 11, 2025 •

edited

Loading

netomi commented Jun 11, 2025 •

edited

Loading