Cannot infer type argument with TypeVarTuple and Callback protocol #17453

Activity

1. cdce8p

   added

   bugmypy got something wrongmypy got something wrong

   topic-pep-646PEP 646 (TypeVarTuple, Unpack)PEP 646 (TypeVarTuple, Unpack)

   topic-protocols

   on Jun 30, 2024
2. ilevkivskyi commented on Jun 30, 2024

   

   ilevkivskyi

   on Jun 30, 2024

   Member

   More actions

   It seems to me you are confusing this with ParamSpec. TypeVarTuple doesn't have any notion of argument kinds, so after you assigned job = Job(action), the type of job is simply Job[str, int]. I guess what you want is this:

   class Job[**P]:
       def __init__(self, target: Callable[P, None]) -> None:
           self.target = target
   
   def run_job[**P](job: Job[P], *args: P.args, **kwargs: P.kwargs) -> None: ...

   Btw with the new syntax underscores are not needed, and not recommended.
3. erictraut commented on Jun 30, 2024

   

   erictraut

   on Jun 30, 2024

   More actions

   @ilevkivskyi, I agree this is a gray area in the typing spec. We should collectively decide whether this is something that should be supported for TypeVarTuples.

   I managed to support the above code in pyright (inspired in part by your work in mypy), but we may want to ultimately disallow this in the spec. I don't have a strong opinion one way or the other.

   In the off chance that this is helpful, I'll explain how I implemented this in pyright. I'm not that familiar with mypy's internals, so I don't know if the approach I used in pyright is feasible for you. When pyright records the specialized type arguments for a tuple, it stores not only the type of the type argument but also a flag is_unbounded that indicates whether the entry is unbounded (i.e. followed by a ...). To support the capture of signatures with default arguments by a TypeVarTuple, I added a second flag is_optional that indicates whether the type argument is required or optional. In effect, I'm storing a bit of information about the parameter that was captured by the TypeVarTuple. When specializing Job above, the second type argument (with type int) is marked as "optional". With this information, I'm able to support parameters with default values when later expanding the specialized TypeVarTuple value for *args: *Ts.

   Code sample in pyright playground

   from typing import Callable, Protocol
   
   class ActionType(Protocol):
       def __call__(self, var: str, context: int = 2) -> None: ...
   
   class Job[*_Ts]:
       def __init__(self, target: Callable[[*_Ts], None]) -> None:
           self.target = target
   
   def run_job[*_Ts](job: Job[*_Ts], *args: *_Ts) -> None: ...
   
   def a1(action: ActionType) -> None:
       job = Job(action)
       run_job(job, "Hello")  # OK
       run_job(job, "Hello", 1)  # OK
       run_job(job, "Hello", "1")  # Type error
4. cdce8p commented on Jul 2, 2024

   

   cdce8p

   on Jul 2, 2024

   CollaboratorAuthor

   More actions

   It seems to me you are confusing this with ParamSpec.

   No. I can't use ParamSpec in that case unfortunately. The function only accepts *args so I need to use TypeVarTuple for it.

   TypeVarTuple doesn't have any notion of argument kinds, so after you assigned job = Job(action), the type of job is simply Job[str, int].

   Strictly speaking, you're correct. However as Eric already pointed out, you implemented something like this for Callables already. The run_job_2 example works with mypy even though that also depends on default arguments.

   I agree this is a gray area in the typing spec. We should collectively decide whether this is something that should be supported for TypeVarTuples.

   I managed to support the above code in pyright (inspired in part by your work in mypy), but we may want to ultimately disallow this in the spec. I don't have a strong opinion one way or the other.

   I'm strongly in favor of keeping the existing support for default arguments. Without that the usefulness of TypeVarTuples for Callable and *args typing would be severely limited. There are quite a few functions (especially in the asyncio module) that already depend on that.
   python/typeshed#11015
   https://peps.python.org/pep-0646/#type-variable-tuples-with-callable
5. ilevkivskyi commented on Jul 2, 2024

   

   ilevkivskyi

   on Jul 2, 2024

   Member

   More actions

   you implemented something like this for Callables already.

   No, I didn't. ActionType is simply a subtype of Callable[[str], None], and callables were always put in the second pass of inference, this is why example two works.

   I know the callables in Python are a mess. I would personally prefer if they were all like def fn(x, y, /, *, z, t) (~ the shell semantics that is already familiar to everyone), but we are where we are. Anyway, even in such ideal world I don't see how this can be consistently/intuitively supported. For example:

   class ActionType(Protocol):
       def __call__(self, var: str, context: int = ...) -> None: ...
   
   class Job[*Ts]:
       attr: tuple[*Ts]
       def __init__(self, target: Callable[[*Ts], None]) -> None: ...
   
   action: ActionType
   job = Job(action)
   reveal_type(job.attr)  # What should this be? tuple[str], tuple[str, int], or tuple[str] | tuple[str, int]?

   Of course this is an artificial example, but it shows how we quickly get into the formal kind vs actual kind confusion again. Btw I did implement ~similar horrible special-casing for ParamSpec (even going beyond/against what is in the PEP), but ParamSpec is already inherently ad-hoc and is strongly tied to callables. I want to keep TypeVarTuple more "generic" (i.e. not tied to callables semantics).