Introduce typing.STRICTER_STUBS

@AlexWaygood

Some functions can return different types depending on passed arguments. For example:

open(name, 'rb') returns io.BufferedReader, whereas
open(name, 'wb') returns io.BufferedWriter.

The typeshed accurately models this using @typing.overload and typing.Literal. However there is the case that the argument value deciding the return type cannot be determined statically, for example:

def my_open(name: str, write: bool):
    with open(name, 'wb' if write else 'rb') as f:
        content = f.read()

In this case typeshed currently just claims that open returns typing.IO[Any], so content ends up having the type Any, resulting in a loss of type safety (e.g. content.startswith('hello') will lead to a runtime error if the file was opened in binary mode, but type checkers won't be able to warn you about this because of Any).

While typeshed could theoretically just change the return type to typing.IO[Union[str, bytes]], that would force all existing code bases that currently rely on Any to type check to update their code, which is of course unacceptable.

When starting a new project I however want the strictest type stubs possible. I explicitly do not want standard library functions to return unsafe values like Any (or the a bit less unsafe AnyOf suggested in #566), when the return types can be modeled by a Union.

I therefore propose the introduction of a new variable typing.STRICTER_STUBS: bool, that's only available during type checking.

Which would allow typeshed to do the following:

if typing.STRICTER_STUBS:
    AnyOrUnion = typing.Union
else:
    AnyOrUnion = typing.Any

Ambiguous return types could then be annotated as e.g. -> typing.IO[AnyOrUnion[str, bytes]].

This would allow users to opt into stricter type stubs, if they so desire, without forcing changes on existing code bases.

CC: @AlexWaygood, @JelleZijlstra, @srittau, @hauntsaninja, @rchen152, @erictraut

P.S. Since I have seen Union return types being dismissed because "the caller needs to use isinstance()", I want to note that this is not true, if the caller wants to trade type safety for performance, they can always just add an explicit Any annotation to circumvent the runtime overhead of isinstance. Union return types force you to either handle potential type errors or explicitly opt out of type safety, which I find strongly preferable to lack of type safety by default.

Interesting idea. How would users opt into the "stricter" stubs? Do you propose that type checkers add a new config setting?

I'm not a fan of reusing the TYPE_CHECKING constant, by the way -- I'd prefer to see a new constant introduced in typing.py. But that's a minor point.

Yes, type checkers would need to introduce a new config setting. I'd propose --stricter-stubs as a uniform command-line flag.

I'm not a fan of reusing the TYPE_CHECKING constant, by the way

Yes, good point. I updated the proposal to typing.STRICTER_STUBS with a boolean value. It would actually only need to be available during type checking, so no change to typing.py should be necessary (but maybe it's still a good idea?).

Here are two other potential solutions to this problem.

Formally introduce an Unknown type and use it in place of Any in these cases. Pyright internally tracks an Unknown as a form of Any except that in "strict" mode, it will complain about the use of Unknown. This idea is borrowed from TypeScript, which formally exposes an unknown type.
Introduce a OneOf type an an unsafe union — one where only one subtype needs to be compatible rather than all subtypes. This has been discussed previously but didn't gain enough support to lead to a PEP.

Of these three options, I think the OneOf provides the most utility because it provides much more information for language servers. For example, it allows good completion suggestions to be presented to users. It can also be flagged as unsafe in the strictest modes.

Why do we need a flag? In the OP's example it seems a pretty good idea to change the typechecker behavior to return io.BufferedReader | io.BufferedWriter. (A better example would actually be open(name, 'rb' if binary else 'r'), which could return io.BufferedReader | io.TextIO.)

@gvanrossum

@gvanrossum The type checker does not know about the return types of open. These are only encoded in the type stubs. In this case the stdlib stubs of typeshed. There are currently many projects that rely on Any return types of typeshed, changing the return type in typeshed would break the type checking for these projects. Hence the flag.

@erictraut thanks for bringing up these alternatives. I think the appeal of the flag is that you can set it and forget it, whereas the introduction of a new type would require Python programmers to learn about the type ... and for what? I personally wouldn't want to deal with some functions returning Union and some functions returning AnyOf (besides I find the latter very confusing semantically because the definition of a union is that it can be any one of its arguments, having both AnyOf and Union is bound to result in confusion). Yes OneOf would allow richer LSP autocompletion, however I am not sure if providing autocompletion is such a good idea if the suggestions might lead to type errors during runtime.

@not-my-profile

changing the return type in typeshed would break the type checking for these projects

Are you sure? It may be worth checking whether that's really common in real code. mypy-primer should help.

I like the OneOf idea. Type checkers could add a flag that makes the checker treat the type like a regular Union, which would provide the type safety @not-my-profile asks for.

@gvanrossum

@gvanrossum The type checker does not know about the return types of open. These are only encoded in the type stubs. In this case the stdlib stubs of typeshed.

I'm well aware. :-)

My actual proposal was actually more complex. If we have stubs containing e.g.

@overload
def foo(x: int) -> list[int]: ...
@overload
def foo(y: str) -> list[str]: ...

and we have a call like this

def f(flag: bool):
    a = foo(0 if flag else "")
    reveal_type(a)

then the revealed type could be list[int] | list[str]. But this would require the inferred type of 0 if flag else "" to be int | str, whereas ATM (in mypy at least) the inferred type there is object, being the nearest common (named) type in the MROs of int and str.

There are currently many projects that rely on Any return types of typeshed, changing the return type in typeshed would break the type checking for these projects. Hence the flag.

But in this example the return type is not Any. The call in my example is rejected; however if we help the type checker by forcing a union it will actually do the right thing:

from typing import overload

@overload
def foo(a: int) -> list[int]: ...

@overload
def foo(a: str) -> list[str]: ...

def foo(a): pass

def f(flag: bool):
    a: int | str = 0 if flag else ""
    reveal_type(a)  # int | str
    b = foo(a)
    reveal_type(b)  # list[int] | list[str]

When I tried the OP's example it seems that because the mode expression's type is inferred as str the fallback overload is used, and that's typing.IO[Any]. if I try to help a bit by adding an explicit type, like this:

from typing import Literal

def my_open(name: str, write: bool):
    mode: Literal['rb', 'wb'] = 'wb' if write else 'rb'
    with open(name, mode) as f:
        reveal_type(f)  # typing.IO[Any]

I still get the fallback. This seems to be due to some bug (?) in the typeshed stubs -- if I add buffering=0 it gets the type right:

def my_open(name: str, write: bool):
    mode: Literal['rb', 'wb'] = 'wb' if write else 'rb'
    with open(name, mode, buffering=0) as f:
        reveal_type(f)  # io.FileIO

(@JelleZijlstra @srittau Do you think the former result is a bug in the stubs? Those overloads have buffering: int without a default.)

@not-my-profile

I like the OneOf idea. Type checkers could add a flag that makes the checker treat the type like a regular Union, which would provide the type safety @not-my-profile asks for.

Exactly what I was thinking. #566 (AnyOf/OneOf) would work very well with a strictness check and we could get rid of those pesky Any return typesthat no one likes.

Are you sure? It may be worth checking whether that's really common in real code. mypy-primer should help.

I am unsure how representative 100 projects can be for the far larger Python ecosystem. Anyway I tried it with python/typeshed#7416 and it interestingly enough resulted in some INTERNAL ERRORs for mypy.

(AnyOf/OneOf) would work very well with a strictness check

When should a function return Union and when should a function return OneOf?

Exactly what I was thinking. #566 (AnyOf/OneOf) would work very well with a strictness check and we could get rid of those pesky Any return typesthat no one likes.

But that's not what's going on in the OP's example (see my message).

@gvanrossum

@gvanrossum It's late here and my brain might be a bit mushy. But aren't those orthogonal issues? AnyOf/OneOf can be useful generally in stubs in many situations. And there seems to be a problem with the way type checkers infer a base type instead of a union type in some situations.

Also if I remember correctly, there are some issues with open() when using mypy, since the latest version still has a plug for it, which overrides the stubs for open().

Edit: I will try to give more coherent thoughts tomorrow.

IIRC AnyOf has been proposed before but always met fierce resistance from mypy core devs.

I just wanted to get to the root of the motivation of the issue as presented by the OP, and came up with some interesting (to me) facts.

The plugin for open has (finally) been deleted from the mypy sources (python/mypy#9275) but that was only three weeks ago, so it's probably not yet in the latest release (0.931). The results for my examples are the same on the master branch and with 0.931 though.

Improve open overloads when mode is a literal union

class _WeakUnion(Generic[*_Ts], Any): ...

Afaik the unpack operator in subscript requires Python 3.11 which is a problem because mypy parses the source code with ast.parse, so I guess that means TypeVarTuple is out of the question?

I think we could just do something simple like:

_WeakUnion = Any

instead, since Any can be subscripted arbitrarily. While this does mean that type checkers wanting to support _WeakUnion would have to check the type name instead of the type type, I assume that type checkers already track the type names.

Edit: Apparently mypy doesn't like arguments given to type aliases (unlike pyright which doesn't complain).

class _WeakUnion(Generic[*_Ts], Any): ...

Afaik the unpack operator in subscript requires Python 3.11 which is a problem because mypy parses the source code with ast.parse, so I guess that means TypeVarTuple is out of the question?

We can use typing_extensions.Unpack instead.

Edit: Apparently mypy doesn't like arguments given to type aliases (unlike pyright which doesn't complain).

No, the issue is that Any is not subscriptable, so mypy is correctly raising an error in your python/typeshed#7437 PR.

>>> from typing import Any
>>> Any[str]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Alex\AppData\Local\Programs\Python\Python310\lib\typing.py", line 311, in inner
    return func(*args, **kwds)
  File "C:\Users\Alex\AppData\Local\Programs\Python\Python310\lib\typing.py", line 402, in __getitem__
    return self._getitem(self, parameters)
  File "C:\Users\Alex\AppData\Local\Programs\Python\Python310\lib\typing.py", line 424, in Any
    raise TypeError(f"{self} is not subscriptable")
TypeError: typing.Any is not subscriptable

This feels a lot like feature flags with current any_of direction we're doing. Specifically it reminds me of this comment. I would be very interested in feature flags for new peps in general or features that are useful, but vary across type checkers (recursive types is only one in mind for me).

Main two I'm interested in currently are PEP 646 and recursive types, although I think having new flag for each new pep that introduces new types would allow experimentation/stub improvement faster for type checkers that support them earlier. Both would be useful to try in stubs but without feature flags my guess is it'll take a while for 646 given it's complexity. If we had feature flags we could do in a stub file in typeshed,

if typing.flags.any_of:
  from typing_extensions import AnyOf
  AnyOf = AnyOf
else:
  AnyOf: Any (or some other definition)

or for recursive types,

BasicJSON = str | float | None
if typing.flags.recursive:
  JSON = Mapping[str, JSON] | Sequence[JSON] | BasicJSON
else:
  JSON = Mapping[str, BasicJSON] | Sequence[BasicJSON] | BasicJSON | Mapping[str, Any] | Sequence[Any]

This way type checkers who understand recursion get a fully accurate type, while other type checks still have a reasonable approximation.

Improve open overloads when mode is a literal union (#7428)

class _WeakUnion(Generic[*_Ts], Any): ...

This looks promising. This still is missing the ability to specify a non-Any type as the "fallback" type. For example, static type checkers could prefer str | Any, while language servers might use str | None. We could have an alternative form that supports an explicit fallback type:

class _WeakUnionFallback(Generic[_T, *_Ts], Any): ...

Now type checkers could treat _WeakUnionFallback[str | Any, str, None] as equivalent to str | Any, while language servers could understand it as _WeakUnion[str, None].

If we don't want to wait until PEP 646 is supported everywhere, perhaps we could just use a union type type argument, which would be interpreted as a "weak union"? So instead of writing _WeakUnion[str, None], we'd express it as _WeakUnion[str | None]. Also, instead of _WeakUnionFallback[str | Any, str, None], we'd have _WeakUnionFallback[str | Any, str | None].

I like your idea of using a union form within _WeakUnion. That's a clever way to eliminate the dependency on PEP 646.

I don't see the need for an explicit fallback. Perhaps you could give a concrete example where this would be desirable?

I was thinking that the fallback should be based on type checker configuration and capabilities, not specified by the library author.

Pyright is both a language server and a type checker, but it has no "language server mode" where it evaluate types differently. The two pieces of functionality (LS and type checker) are intrinsically tied. Having different modes for LS and type checker would be confusing because type errors would be inconsistent with LS features like "hover text" where hovering over an identifier displays its type.

I was thinking that pyright would interpret _WeakUnion in one of two ways.

In "basic" type checking mode: as an intersection between Any and union U
In "strict" type checking mode: as union U

In both cases, the LS functionality would provide completion suggestions for all of the subtypes in union U.

Maybe in the future I would consider replacing the "basic" mode with a true "weak union" implementation, but as you pointed out, that would be a lot of work, and I'm not sure it's worth it.

I don't see the need for an explicit fallback. Perhaps you could give a concrete example where this would be desirable?

open is a potential example. Currently if the mode is not known statically, the return type is IO[Any]. This seems like the best type we can currently have, and it's clearly better than plain Any. For example, code like this will generate type errors, as expected:

def f(m: str) -> None:
    f = open('x', mode=m)
    f.bad_call()  # "IO[Any]" has no attribute "bad_call"
    "x" + f  # Invalid types "str" and "IO[Any]"

However, tools that properly support _WeakUnion would probably prefer _WeakUnion[TextIOWrapper, BufferedReader]. This would allow a more precise type for code like this, in addition to catching the errors in the earlier example:

def f(m: str) -> None:
    f = open('x', mode=m)
    s = f.read()
    reveal_type(s)  # _WeakUnion[str, bytes] (instead of Any)

Tools that support _WeakUnion could give better completion suggestions for s (e.g. s.startswith). If we'd have IO[Any] as the return type, s would have the type Any, and we can't give very useful completion suggestions.

Here I think that the best compromise for the return type of open would be _WeakUnionFallback[IO[Any], TextIOWrapper | BufferedReader]. This wouldn't make the return type less precise for tools that support weak unions and don't implement completion suggestions.

I was thinking that the fallback should be based on type checker configuration and capabilities, not specified by the library author.

In the above example, we could perhaps invent rules so that IO[Any] would be automatically inferred as the fallback for tools that don't support _WeakUnion properly. I think that this seems too complicated/ad-hoc to reason about, at least for non-expert typeshed contributors, and spelling the fallback out explicitly would increase clarity.

In "basic" type checking mode: as an intersection between Any and union U

Would this result in worse type checking results compared to IO[Any] in the above example? It seems that completion suggestions would be better, but type checking would be compromised, but I guess that depends on the exact semantic of the intersection type. I wonder if an intersection between IO[Any] and TextIOWrapper | BufferedReader would be a more precise type in this "basic" mode?

In "strict" type checking mode: as union U

Mypy could possibly also support this behavior behind a strictness flag.

Thanks for the example. It's clear to me now, and I agree there's utility in adding an explicit fallback — at least in some cases. These are cases where the stub can provide a fallback that's more precise than Any but still not a full union of all possible return types.

Noting that a flag would also provide a way forward for stricter typing of json loads and dumps functions. I don't see how the unsafe union types proposed could help in that case. For future proofing, assuming that there are many places left that we'll want to stricten typing for, perhaps it should rather be a function than a constant?

if typing.feature("strict-json"): ...

Type checkers would implement configuration parameters to enable or disable features.

In a large code-base with lots of legacy Python, the fact that typeshed uses Any in many places is giving quite a lot of grief. Automating the process of adding type ignores is easy for us, and this enables us to gradually improve explicitly disabled type checking, rather than constantly running into much less visible implicitly disabled type checking, caused by Any.

The introduction of a function would also allow third party packages to provide such conditional strictness levels.

@overload

My actual proposal was actually more complex. If we have stubs containing e.g.

@overload
def foo(x: int) -> list[int]: ...
@overload
def foo(y: str) -> list[str]: ...
and we have a call like this

def f(flag: bool):
a = foo(0 if flag else "")
reveal_type(a)
then the revealed type could be list[int] | list[str]. But this would require the inferred type of 0 if flag else "" to be int | str, whereas ATM (in mypy at least) the inferred type there is object, being the nearest common (named) type in the MROs of int and str.

There are currently many projects that rely on Any return types of typeshed, changing the return type in typeshed would break the type checking for these projects. Hence the flag.

But in this example the return type is not Any. The call in my example is rejected; however if we help the type checker by forcing a union it will actually do the right thing:

from typing import overload

@overload
def foo(a: int) -> list[int]: ...

@overload
def foo(a: str) -> list[str]: ...

def foo(a): pass

def f(flag: bool):
a: int | str = 0 if flag else ""
reveal_type(a) # int | str
b = foo(a)
reveal_type(b) # list[int] | list[str]

From Mypy 1.12, the type of a is int | str and the type of b is list[int] | list[str], same on pyright
The same works for a = "r" if flag else "w" in pyright, but on mypy you need to declare that a has type Literal["w", "r"], because mypy is more conservative infering literals

How can I opt out of MaybeNone? In the years of Python typing, I never wanted to see an Any anywhere, and always wanted precise typing.

Navigation Menu

Repository navigation

Introduce `typing.STRICTER_STUBS` #1096

18 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Introduce typing.STRICTER_STUBS #1096

Description

Activity

AlexWaygood commented on Mar 1, 2022

not-my-profile commented on Mar 1, 2022

erictraut commented on Mar 1, 2022

gvanrossum commented on Mar 1, 2022

not-my-profile commented on Mar 1, 2022

JelleZijlstra commented on Mar 1, 2022

gvanrossum commented on Mar 1, 2022

srittau commented on Mar 1, 2022

not-my-profile commented on Mar 1, 2022

gvanrossum commented on Mar 1, 2022

srittau commented on Mar 1, 2022

gvanrossum commented on Mar 1, 2022

18 remaining items

not-my-profile commented on Mar 4, 2022

JelleZijlstra commented on Mar 4, 2022

AlexWaygood commented on Mar 4, 2022

hmc-cs-mdrissi commented on Mar 4, 2022

JukkaL commented on Mar 7, 2022

erictraut commented on Mar 7, 2022

JukkaL commented on Mar 8, 2022

erictraut commented on Mar 8, 2022

antonagestam commented on Mar 12, 2024

antonagestam commented on Mar 12, 2024

Polandia94 commented on Aug 2, 2025

flying-sheep commented on Dec 26, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Introduce `typing.STRICTER_STUBS` #1096