Types not added to binder on initial assignment

from typing import Union
x = 0  # type: Union[int, str]
reveal_type(x)  # Revealed type is 'Union[builtins.int, builtins.str]'
x = 0
reveal_type(x)  # Revealed type is 'builtins.int'

This means if you want to declare a union and then immediately use it as one specific member, you'll have an awkward time. For example:

x = "foo" # type: Optional[str]
x + "bar"  # E: Unsupported operand types for + ("Union[str, None]" and "str")

To be consistent with other assignments, we should add types to the binder on initial assignment. I think a number of Union/Optional tests depend on the current behavior, so those will need to be updated.

(I came across this issue while working on #1955.)

I also noticed this while working on the binder and wondered whether it was intentional. It seems unlikely that any real code relies on the current behavior, but as you note, probably most of the binder-related tests do, if they are to test what was intended. I wonder what a good replacement for the tests is, especially in a post-strict-optional world. Maybe add a generic function

T = TypeVar('T')
def something() -> T: pass

to the test fixtures and then replace initializations like

x = 1  # type: Union[int, str]

by

x = something()  # type: Union[int, str]

Making this change is pretty easy, but it causes 2 new errors when running mypy on mypy.
The first boils down to this:

class Super: pass
class Sub(Super): pass

def f() -> List[Super]:
    x = Sub()  # type: Super
    ret = [x]
    return ret  # E: Incompatible return value type (got List[Sub], expected List[Super])

This can be worked around by giving ret an explicit type, but still seems problematic and a potential source of confusion.

The second boils down to this:

y = [1]
x = y  # type: List[Any]
x.append("foo")  # E: Argument 1 to "append" of "list" has incompatible type "str"; expected "int"

This one should arguably be a cast instead (and that would be a reasonable workaround).

Maybe the conclusion is that we should only do this if the specified type is a Union?

Just doing this for unions sounds reasonable to me. More generally, this would likely be useful for conditional definitions, and unions are commonly used for conditional definitions.

For non-unions, this could be reasonable, but it would be much harder to support without breaking the examples given above:

if <cond>:
    e = Manager()  # type: Employee
    e.direct_reports = ...    # Only works for Manager, not Employee
else:
    e = Engineer()
    ...

I had a long diatribe about this but changed my mind several times while writing, so I guess it's complicated. However I don't see how changing this would break the last Example Jukka gave?

If we only make this change for unions, it would likely not break a lot of code. My example doesn't work right now, I think, so it's already a problem (but likely not a major one, so we can continue ignoring it).

Oooh, we definitely need to talk!

…

--Guido (mobile)

But the proposed change would fix that example wouldn't it?

Yes.

(If we'd always adds types to binder, this would fix my example. If we'd only do this for union types then it wouldn't fix it.)

So then why shouldn't we just fix it the way David originally proposed, always adding the type to the binder?

Adding it always to the binder breaks some other things, such ones mentioned by David in #2008 (comment).

There are other things it could break. For example, what if we want to make a collection not support mutation:

x = [1, 2, 3]  # type: Sequence[int]    # Don't want anybody to modify this
# I wouldn't expect type of x to be List[int] here
x.append(4)  # This should be an error

Not being able to override the annotated type breaks some pretty basic use cases, and it can be surprising. For union types it is less of a problem, as they are usually used in very specific ways, and they are relatively rare.

All in all this looks like a pretty complex issue to me (though the implementation would be straightforward).

I don't like either 2 or 5, they seem a big deviation from the status quo for non-definition assignments.

In #3178 , to solve that and several other issues, I was suggesting to consider using ranges of types (limited by upper and lower bound) rather than precise types in the inference engine. If we move down that path (and I owe a more detailed explanation / proof that it actually helps), it seems it might help solve this problem as well.

In particular, if the variable can be bound to a range of types rather than a precise type, proposal (3) would no longer have problematic interaction with invariant / contravariant generics: the type checker will make sure the type range overlaps with what the context requires.

There is a problem with 4, these two forms should be invariant:

  x: Sequence[int] = [0]

  y: Sequence[int]
  y = [0]

This is written in PEP 526 for a reason. At the time of discussion on python-dev there was some (quite strong) opposition to making these two forms non-equivalent. And note that PEP affects not only mypy but also pytype, PyCharm, etc.

I think I like 3 the most. I find it conceptually simple: a declared type sets an initial bound on type, but we could locally make this bound more narrow depending on context, even immediately by the r.h.s. in first assignment.

Yeah, I've always been unwavering in favor of #3. But #4 is halfway there, and in addition to easy of implementation and concerns about causing tons of new errors, there is a certain amount of compromise between the members of the Dropbox mypy team here (as you can read between the lines of recent comments).

@ddfisher

@ddfisher Yes, my thinking on this has changed. First, I think that we overestimated the user confusion proposal 4 would cause. Second, the cost of migration to 2 is now much higher than it was during the original discussion and harder to justify. Third, proposal 5 feels too ad hoc and too different from the current way.

@pkch Can you provide code examples that show how #3178 would help with this issue?

@ilevkivskyi @gvanrossum I tentatively agree that 3 might be the best long-term approach, but 4 should be easier to implement, and it's closer to 3 than what mypy currently does. It should also be easier to migrate to than 3. We should still be able to move to 3 in the future, and if you can come up with a simple implementation approach for 3 that deals with Any types in a good way, then I might be happy to skip 4 altogether.

I now suspect my idea is too complex to be feasible. I was thinking to keep, for each variable, the range of types: the lower bound is the type from the binder, and the upper bound is the type declared in the annotation. I wanted mypy to use this range (rather than just the precise type from the binder) to
see if the variable can be used in any given context.

However, I found that the types become too complex if the type parameters can be type ranges.

Anyway, let me comment on the proposal 3.

My basic mental model is that the type information lives in a layer above the code, and imposes constraints on the code that mypy will validate. Type inference is a very intuitive refinement of that: mypy will reduce the burden on the developer by automatically generating type information if the developer didn't provide it.

I was hoping type binding is only a slightly more complex refinement: mypy may reduce false positives by automatically relaxing type constraints if it concludes that the code is still safe with the weaker constraints.

Even if it followed that principle, type binding would be controversial, since it gives the source code priority over explicit type annotations, and therefore misses some bugs:

x = [0]  # type: Sequence[int]
x[0] = 1  # it may be a bug, but mypy won't catch it

But at least it's moving mypy in a single direction (less strict), and it's a reasonable cost to pay in order to support many common idioms in existing codebases.

But proposal 3 sometimes makes mypy more strict, and that feels very counterintuitive (see example with invariance/contravariance). Instead of using information from the code to relax type constraints, mypy unexpectedly adds additional constraints to those provided by the developer.

That said, the problem already exists:

# current master
class A: pass
class B(A): pass
def f(List[A]) -> None: ...

x = B()  # type: A
x = B()
lst = [x]
f(lst)  # E: expected List[A], got List[B]

I can't think of a simple solution. Rechecking the entire function/module without using type bindings if a type error is detected is too weird and will cause more surprises. And doing something more granular seems very expensive, as my type range attempts convinced me.

Could we consider this behavior? It seems it can get rid of several downsides of proposal 3:

x = [0]  # type: Sequence[int]
reveal_type(x)  # Revealed type is 'Sequence[int]'; temporarily bound to 'List[int]'
x[0] = 1  # this is allowed because x is temporarily bound to 'List[int]'

These are the problems that it solves:

User confusion: after x = ... # type: Foo, x may not have type Foo.

Foo will always show up in revealed type as the original type, so no more user confusion.

Bad interaction with invariant/contravariant Generics:

This would work fine now:

class A: pass
class B(A): pass
def f() -> List[A]:
    x = B()  # type: A
    reveal_type(x)  # Revealed type is 'A'; temporarily bound to 'B'
    lst = [x]
    reveal_type(lst)  # Revealed type is 'List[A]', temporarily bound to 'List[B]'
    # next line is ok because mypy will try both the original type and the temporarily bound type
    # as long as at least one of them works, it will not complain
    return lst

Unclear interactions with Any types. What should the bound type of x be in this example?

Now something like this would probably be acceptable:

x = [1] # type: Sequence[Any]
reveal_type(x)  # Revealed type is 'Sequence[Any]'; temporarily bound to 'List[int]'
x[0] = 1  # ok, using temporary binding List[int]
x[0] = 0.5  # type error
x[0] > 'abc'  # ok, using original type Sequence[Any]

Conceptually, this makes type annotations more like type hints. However, each variable can have only one type annotation on it (and this is enshrined in PEP 526 — the new variable type declaration syntax for Python 3.6), so this is a bit awkward.

This is no longer the case because type annotation is always respected in the form of the unchangeable "original" type.

This is the only downside that is not fixed:

No way to be specific about locally allowed operations via explicit types:

x = [0]  # type: Sequence[int]
x[0] = 1  # this is allowed even though Sequence is immutable!

But that is unsurprising because the entire approach of type binding comes at the known cost: we trust the code enough to occasionally overrule the type annotation.

Decreasing the priority of this since the current behavior seems to be mostly fine and I haven't heard many complaints about this in a while. I'm actually leaning towards just keeping the current behavior.

In Python 3.6 code the binder behavior can be more easily tweaked by annotating the variable before initialization:

x: Optional[int]
x = 1
reveal_type(x)  # int

I've started a discussion about this in typing-sig.

@erictraut

In the typing-sig discussion, @erictraut argued convincingly that type checkers should follow what PEP 526 says, and Pyre's @shannonzhu agreed, so I think we have to follow suit.

It should make no difference whether the assignment is combined with the declaration or separate. IOW, we should implement Proposal 3.

(PS. Everyone also agreed that narrowing should not occur if the declared type is Any. Mypy does this already. There is some discussion possible about whether it should occur for types containing Any, but how to narrow is a different discussion anyways.)

There can be some unintentional? edge cases with special cased types such as tuple. Code sample in pyright playground:

from typing import Protocol, Self, overload, SupportsIndex

class SupportsSlicing[T_co](Protocol):
    @overload
    def __getitem__(self, index: SupportsIndex, /) -> T_co: ...
    @overload
    def __getitem__(self, index: slice, /) -> Self: ...

lst: list[int] = [1,2,3,4]
x: SupportsSlicing[int] = lst  # ✅ OK
tup: tuple[int, ...] = (1, 2, 3, 4)
y: SupportsSlicing[int] = tup  # ❌ type error

This is because the inferred type is tuple[Literal[1], Literal[2], Literal[3], Literal[4]].

Navigation Menu

Repository navigation

Types not added to binder on initial assignment #2008

17 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Types not added to binder on initial assignment #2008

Description

Activity

ddfisher commented on Aug 10, 2016

rwbarton commented on Aug 10, 2016

ddfisher commented on Oct 25, 2016

JukkaL commented on Oct 25, 2016

gvanrossum commented on Oct 25, 2016

JukkaL commented on Oct 25, 2016

gvanrossum commented on Oct 25, 2016

gvanrossum commented on Oct 25, 2016

JukkaL commented on Oct 26, 2016

JukkaL commented on Oct 26, 2016

gvanrossum commented on Oct 26, 2016

JukkaL commented on Oct 26, 2016

17 remaining items

gvanrossum commented on Apr 21, 2017

pkch commented on Apr 21, 2017

ilevkivskyi commented on Apr 21, 2017

gvanrossum commented on Apr 21, 2017

JukkaL commented on Apr 22, 2017

pkch commented on Apr 24, 2017

pkch commented on May 3, 2017

JukkaL commented on Sep 18, 2017

gvanrossum commented on Sep 2, 2020

gvanrossum commented on Sep 26, 2020

randolf-scholz commented on May 13, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions