Unexpected type binding when wrapping functions #12919

Activity

1. mailund

   added

   bugmypy got something wrongmypy got something wrong

   on Jun 1, 2022
2. erictraut commented on Jun 1, 2022

   

   erictraut

   on Jun 1, 2022

   More actions

   Type variables are "solved" in the context of a call expression that targets the constructor of a generic class or a generic function. Only the type variables in the scope of that class or function are solved. Once a type variable goes out of scope, it can no longer be solved.

   The expression listed(g) causes mypy to solve the TypeVar _T@lifted (i.e. _T in the scope of class lifted). The solution in this case is _T@g, which means that listed(g) evaluates to type lifted[_T@g]. However, at this point, _T@g is no longer in scope. It's an unsolved TypeVar from the scope of function g. It is not associated with the lifted class or its __call__ method, which means it cannot be solved in subsequent call expressions like lifted(g)(1).

   FWIW, pyright largely agrees with mypy in this case.

   So I don't think this is a bug in mypy. It's expected behavior.
3. mailund commented on Jun 1, 2022

   

   mailund

   on Jun 1, 2022

   Author

   More actions

   Ok, I think I understand what you are saying, @erictraut, but let me just check. If I use a generic function or class with a type variable, I bind it when I apply it, and it is no longer functioning as a type variable. If I want the wrapped function to work, the wrapping has to know the concrete type.

   So, for example, with a curry function (as a Generic)

   from typing import (
       TypeVar, ParamSpec,
       Callable as Fn,
       Optional as Opt,
       Generic,
   )
   
   _T = TypeVar('_T')
   
   class curry(Generic[_T]):
       def __init__(self, f: Fn[[_T, _T], _T]) -> None:
           self._f = f
       def __call__(self, x: _T) -> Fn[[_T], _T]:
           return lambda y: self._f(x, y)

   I can't do something like

   @curry
   def f(x: _T, __y: _T) -> _T:
       return x

   because _T isn't free after wrapping, and I cannot do this either

   @curry[_T]
   def f(x: _T, __y: _T) -> _T:
       return x

   either, because I can't use a type variable like this when it is unbound?

   I can, however, use it with concrete types

   def f(x: _T, __y: _T) -> _T:
       return x
   
   fi = curry[int](f)
   reveal_type(fi)                  # curry[int]
   reveal_type(fi(1))               # def int -> int
   reveal_type(fi(1)(2))            # int
   
   fs = curry[str](f)
   reveal_type(fs)                  # curry[str]
   reveal_type(fs("foo"))           # str -> str
   reveal_type(fs("foo")("bar"))    # str

   or I can use _T inside a function that uses the type variable because there it will be bound when I evaluate the function and thus get a concrete type there.

   def bind(x: _T, f: Fn[[_T,_T], _T]) -> Fn[[_T], _T]:
       # ok here since _T is bound when we evaluate
       res = curry[_T](f)
       return res(x)
       
   b = bind(12, f)
   reveal_type(b)  # int -> int
   
   b2 = bind("foo", f)
   reveal_type(b2)  # str -> str

   As long as I don't use an unbound type variable when instantiating the Generic here, it seems to work. Is that guaranteed, and is that the way to achieve what I tried above?
4. sterliakov commented on Jun 2, 2022

   

   sterliakov

   on Jun 2, 2022

   Collaborator

   More actions

   Just adding my 2 cents here, because I've faced this problem once before (and ended with very ugly pattern that violates DRY, see below).

   Related: #1317

   It is understandable why does mypy "solve" this variable earlier than we want. But there is no way to define the following decorator: "given function that takes fixed arguments set (say, f(x, y) that implements min signature, like f(x: _T, y: _T) -> _T), produce a function that has same signature with types Transformed[_T]", where Transformed is expressible with typing primitives and depends only on _T (propagating Optional is just one of possible use cases). It is important that _T is the same for decorated function and result, but we do not know _T value in advance.

   I think that the following (very raw) approach should be better:

   Given decorator:

   from typing import Callable, SupportsInt, TypeVar
   
   _T = TypeVar('_T')
   _P = TypeVar('_P', bound=SupportsInt)
   
   def allow_none(f: Callable[[_T], _T]) -> Callable[[_T | None], _T | None]:
       def inner(x: _T | None, /) -> _T | None:
           return None if x is None else f(x)
       return inner
   
   @allow_none
   def foo(x: _P) -> _P:
       if x is None:
           raise ValueError
       return x

   After producing F[_P | None] type checker should recognize the fact that _P is used both on LHS and RHS of Callable definition. When it happens, _P should go back to F scope, allowing the expected substitution with respect to SupportsInt bound.

   The same should happen is resulting type is Generic or Protocol: instead of producing generic class with unsolvable type variable, mypy should keep it unresolved. Now any function that performs similar operation (f(x: G[_T]) -> G[_T | None], where G is any Protocol or Generic) is malformed: when applied to generic with type variable G[_T], it produces G[_T | None] with _T out of scope, so only Any and None are compatible. It is definitely not the desired behavior.

   I suppose that no existing code should be broken after such change. The resolution algorithm now produces unusable functions (they don't take expected arguments), so usage of such decorators must be very rare and complex. When decorated function is using concrete types, this approach will change nothing.

   The only solution now is to explicitly re-declare type of decorated function. But why are they compatible, if foo(1) is OK while allow_none(_foo)(1) throws mypy error: E: Argument 1 has incompatible type "int"; expected "Optional[_P]"? I don't know, it seems to be inconsistent, LHS type is wider like x: int = 0.1, isn't it?

   ...  # See example above
   
   def _foo(x: _P) -> _P:
       if x is None:
           raise ValueError
       return x
   
   foo: Callable[[_P | None], _P | None] = allow_none(_foo)
   
   reveal_type(foo)  # N: Revealed type is "def [_P <: typing.SupportsInt] (Union[_P`-1, None]) -> Union[_P`-1, None]"
   reveal_type(foo(None))  # N: Revealed type is "None"
   reveal_type(foo(1))  # N: Revealed type is "Union[builtins.int, None]"

   👍React with 👍1Reacted by Thomas Mailund
5. ilevkivskyi

   closed this as completedin #15287on Jun 18, 2023
6. ilevkivskyi

   added a commit that references this issue on Jun 18, 2023

   Foundations for non-linear solver and polymorphic application (#15287)

   ...

   This issue will close once commit 0873230 is merged into the 'master' branch.

   0873230