PEP 612 tracking issue

Hello!

I got a little scared off from working on this by the many other enthusiastic people volunteering their time in #8373. However, there doesn't seem to have been much progress; since I have some extra time now, I figured sharing what I've got could kickstart things. Apologies if someone — or whatever a GSoC is — is already working on it :-) Branch is visible at https://github.com/hauntsaninja/mypy/tree/pep612, happy to make it a draft PR if that would help anything.

Overall seems like the plan would be:

Focus on generic functions first, implement generic class support later, as pyre plans to.
Make the fairly straight forward changes in semanal.py and nodes.py so you have the AST. Make relevant visitors visit it (look at where we visit_type_var_expr).
Change typeanal.py, tvar_scope.py, etc to get binding going to the types you create in types.py. Seems like this would be the place to enforce valid use location too.
Implement the semantics of ParameterSpecification in checkexpr.py, infer.py, solve.py, etc.
Add a runtime implementation to mypy_extensions

Since ParameterSpecification is just a juiced up type variable, it makes sense to try and re-use existing logic for TypeVar as much as possible (especially if later PEPs make use of the bounds and variance arguments that PEP 612 reserves). In my branch, I've chosen to add a base class TypeVarLikeX in a couple different class hierarchies, e.g. TypeVarLikeExpr, TypeVarLikeDef. Since I'm not super familiar with mypy's code, this also helps me stumble upon things that I might need to change (it feels great trusting mypy to help improve mypy!). But this is a fairly large refactor (the one-line change in the details snippet alone will require changes in a dozen files), and could maybe affect like plugins or something, so I thought I'd post this before exploring further.

Details

--- a/mypy/types.py
+++ b/mypy/types.py
@@ -983,7 +983,7 @@ class CallableType(FunctionLike):
                  fallback: Instance,
                  name: Optional[str] = None,
                  definition: Optional[SymbolNode] = None,
-                 variables: Optional[List[TypeVarDef]] = None,
+                 variables: Optional[Sequence[TypeVarLikeDef]] = None,

I'm also a little worried about issues like #1317 and #5738. I don't yet know the code well enough to anticipate what issues we might run into, but I'd welcome any thoughts on interactions between existing TypeVar code and any PEP 612 implementation.

Your plan looks good to me!

I don't expect that #1317 or #5738 will be blockers for this, though they may affect some use cases (which likely don't work currently anyway). In particular, decorating generic functions might not work. I'm planning to work on one or both of those issues in the near future, since they are really quite bad and have been open for a long time.

Please don't hesitate to ask further questions here (or email be directly).

I encourage you to start the work on this. Most likely I will just accept the PEP, given that we've seen no further feedback on typing-sig.

Thanks for the kick! I threw some hours at this today, in particular I managed to get through a good amount of the somewhat tedious refactoring caused by generalising TypeVar. I've made a good amount of progress for typeanal of ParamSpec. Branch is still at https://github.com/hauntsaninja/mypy/tree/pep612

Maybe you can submit it as a PR and mark it Draft? I personally find it hard to follow a change like this as a branch in someone else's repo.

Sure, I opened a draft PR here: #9250 All feedback welcome!
Some notes:

I tend to rewrite history a lot on my personal dev branches, but will avoid doing that if people start reviewing things or ask me not to
I expect to be able to get things done on Saturdays, but aside from that am a little short on time. If need be, I'm happy to hand over whatever I've got — just let me know!
As a summary of where thing stand:
- semantic analysis for ParamSpec for generic functions is in a good place, eg testParamSpecLocations in the draft PR passes
- my next step is figuring out the types and type var solving necessary to eg, allow mypy to understand how changes_return_type_to_str would work
- semantic analysis for ParamSpec components, ParamSpec for generic classes and Concatenate mostly still remains

pep612: more semantic analysis for paramspec (#9422)

So how far have you gotten with this? I see two merged PRs but am afraid to look.

One comment: we're currently looking into support for PEP 612 in typing.py for 3.10, and after that in typing_extensions.py for previous Python versions, so it would be nice if these were also supported:

from typing_extensions import ParamSpec, Concatenate
from typing import ParamSpec, Concatenate

We could actually use some feedback on the PR we're working on: the contributor is currently looking into fairly strict checking of the arguments, but I'm wondering if that's useful -- we can't go as far as what you did in mypy_extensions.pyi (ParamSpec = 0, Concatenate = 0 :-) but we could perhaps just accept anything once we see a ParamSpec or Concatenate as part of a type, rather than validating that the usage matches with the PEP (that could be considered mypy's responsibility). Thoughts?

There hasn't been much progress since my last comment; I sort of deprioritised this in favour of working on other mypy issues that seemed more pressing. But what exists should already support both typing_extensions and typing:

mypy/mypy/semanal.py

Line 3109 in 35e5031

call = self.get_typevarlike_declaration(

I very much agree that ParamSpec shouldn't validate much at runtime. I think Callable already validates too much, which will necessitate hacks for the backport of ParamSpec to typing_extensions like https://github.com/facebook/pyre-check/blob/0ba810e07635955cbcdef7cd2ab3722ab01dde5b/pyre_extensions/__init__.py#L75 I'll take a look at that PR soon.

I currently get a traceback when using ParamSpec in code:

version: 0.800+dev.05d9fb15dcec8bde4e7184ca387e7d8acea68792
Traceback (most recent call last):
  File "C:\Users\gvanrossum\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\gvanrossum\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\gvanrossum\mypy\mypy\__main__.py", line 23, in <module>
    console_entry()
  File "C:\Users\gvanrossum\mypy\mypy\__main__.py", line 11, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "C:\Users\gvanrossum\mypy\mypy\main.py", line 90, in main
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "C:\Users\gvanrossum\mypy\mypy\build.py", line 180, in build
    result = _build(
  File "C:\Users\gvanrossum\mypy\mypy\build.py", line 254, in _build
    graph = dispatch(sources, manager, stdout)
  File "C:\Users\gvanrossum\mypy\mypy\build.py", line 2638, in dispatch
    process_graph(graph, manager)
  File "C:\Users\gvanrossum\mypy\mypy\build.py", line 2964, in process_graph
    process_stale_scc(graph, scc, manager)
  File "C:\Users\gvanrossum\mypy\mypy\build.py", line 3062, in process_stale_scc
    graph[id].type_check_first_pass()
  File "C:\Users\gvanrossum\mypy\mypy\build.py", line 2131, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "C:\Users\gvanrossum\mypy\mypy\checker.py", line 294, in check_first_pass
    self.accept(d)
  File "C:\Users\gvanrossum\mypy\mypy\checker.py", line 401, in accept
    stmt.accept(self)
  File "C:\Users\gvanrossum\mypy\mypy\nodes.py", line 777, in accept
    return visitor.visit_decorator(self)
  File "C:\Users\gvanrossum\mypy\mypy\checker.py", line 3582, in visit_decorator
    sig, t2 = self.expr_checker.check_call(dec, [temp],
  File "C:\Users\gvanrossum\mypy\mypy\checkexpr.py", line 911, in check_call
    return self.check_callable_call(callee, args, arg_kinds, context, arg_names,
  File "C:\Users\gvanrossum\mypy\mypy\checkexpr.py", line 996, in check_callable_call
    callee = freshen_function_type_vars(callee)
  File "C:\Users\gvanrossum\mypy\mypy\expandtype.py", line 44, in freshen_function_type_vars
    assert isinstance(v, TypeVarDef)
AssertionError:
t.py:13: : note: use --pdb to drop into pdb

I'd like to make some progress with this -- I don't think I would be very useful with this traceback, but I can do things like adding ParamSpec etc. to typing_extensions (so they can be used with earlier Python versions), both in typeshed and in the actual typing_extensions.py module, and also do a release of the latter (once it's ready).

Huh, there are TODOs all over the code TODO(shantanu): fix for ParamSpecDef and my example hit one. This is a minimal example BTW:

from typing import Callable, ParamSpec, TypeVar

P = ParamSpec("P")
T = TypeVar("T")

def wlog(func: Callable[P, T]) -> Callable[P, T]:
    return func

@wlog
def add(a: int, b: int) -> int:
    return a+b

I'll see if I can do something about it.

(UPDATE: I should have read your update a few comments earlier before reporting a traceback. :-)

Fascinating. I'm hitting the very problem described in PEP 612's Motivation chapter. Would be great to have a solution for this.

@sebastian-philipp

@sebastian-philipp I think #10883 and #10866 and #10862 does enough to let you use ParamSpec in your projects without breaking downstream mypy users, but you'd need to use pyre or another checker to make sure your own usage is valid

well, for now I'd like to stick with mypy. Switching to a different type checker, especially one that doesn't support getting installed via pip is a non-trivial task.

@sebastian-philipp

@sebastian-philipp I think #10883 and #10866 and #10862 does enough to let you use ParamSpec in your projects without breaking downstream mypy users, but you'd need to use pyre or another checker to make sure your own usage is valid

ParamSpec with Generic seems missing

#11362

@NeilGirdhar

@NeilGirdhar, the api_boundary decorator directly returns a Callable whereas this example returns a specialized generic class (SyncToAsync). Pyright and (I presume) mypy special-case Callable when it is used as a return type so unsolved TypeVars can be solved when the callable is invoked even if these TypeVars are no longer in scope. Technically, this is incorrect behavior, but from a practical standpoint, it's important to bend the rules here to support a variety of common use cases. The same accommodation isn't made for arbitrary generic classes. Arguably, it should be applied for callback protocols or for __call__ methods in general, since they act like Callable.

I think at least mypy should complain where the unusable instance is created in addition to when the unusable instance is used:

@sync_to_async  # ParamSpec cannot be used with generic functions
def sleep(delay: float, value: T) -> T:
    time.sleep(delay)
    return value

Technically, this is incorrect behavior, but from a practical standpoint,

Why is it incorrect behavior?

Arguably, it should be applied for callback protocols or for call methods in general, since they act like Callable.

Yes, that makes sense. It seems like it could work in general though—not just with callables. If MyPy ever supports higher-kinded type variables, this kind of functions of generic will need to be supported, I think.

Properly type _generative, decorator, public_factory

I am facing issue with the following code:

from typing import Callable, Generic
from typing_extensions import ParamSpec


P = ParamSpec("P")


class Signal(Generic[P]):
    def __init__(self) -> None:
        pass

    def subscribe(self, f: Callable[P, None]) -> None:
        pass

    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> None:
        pass


class M:
    def __init__(self) -> None:
        self.sig: Signal[[int, str]] = Signal()

    def emit(self) -> None:
        self.sig(1, 2)

Checking it with mypy (0.930 and 0.940+dev.ba0e9d6427fa15a2c3420d4427e88b99514d22f9), Python 3.9 and 3.10, I get the following:

a.py:21: error: Bracketed expression "[...]" is not valid as a type
a.py:21: note: Did you mean "List[...]"?
Found 1 error in 1 file (checked 1 source file)

whereas pyright seems to get it right:

  /home/uken/src/mypy-bug3/a.py:24:21 - error: Argument of type "Literal[2]" cannot be assigned to parameter of type "str" in function "__call__"
    "Literal[2]" is incompatible with "str" (reportGeneralTypeIssues)
1 error, 0 warnings, 0 infos

pyre gets some things right but also emits some strange error (the first one):

a.py:21:8 Incompatible attribute type [8]: Attribute `sig` declared in class `M` has type `Signal[[int, str]]` but is used as type `Signal[a.P]`.
a.py:24:20 Incompatible parameter type [6]: Expected `str` for 2nd positional only parameter to call `Signal.__call__` but got `int`.

P.S.: if Signal class is declared in a separate module, running mypy for the second time fails with the following assertion:

...
  File "venv/lib/python3.9/site-packages/mypy/nodes.py", line 2815, in deserialize
    defn = ClassDef.deserialize(data['defn'])
  File "venv/lib/python3.9/site-packages/mypy/nodes.py", line 1024, in deserialize
    [mypy.types.TypeVarType.deserialize(v) for v in data['type_vars']],
  File "venv/lib/python3.9/site-packages/mypy/nodes.py", line 1024, in <listcomp>
    [mypy.types.TypeVarType.deserialize(v) for v in data['type_vars']],
  File "venv/lib/python3.9/site-packages/mypy/types.py", line 455, in deserialize
    assert data['.class'] == 'TypeVarType'
AssertionError

which keeps on occurring until removal of mypy's cache entry corresponding to the file containing the Signal class.

Hi, I face issues, when using ParamSpec.args as a tuple and ParamSpec.kwargs as a dict

from typing_extensions import ParamSpec
from typing import Callable

P = ParamSpec("P")

def foo(func: Callable[P, None]):
    def _inner(*args: P.args, **kwargs: P.kwargs):
        x = kwargs.pop("x")

Error: testscript.py:10: error: "P.kwargs" has no attribute "pop"

The code works at runtime. Not sure if it's part of the PEP, that this is possible

I opened #12386 for that

We also have topic-paramspec PEP 612, ParamSpec, Concatenate now for listing remaining bugs with ParamSpec.

Basic ParamSpec Concatenate and literal support (#11847)

I believe, ParamSpec is now fully implemented in mypy (minus potential bugs)?

#11847 made a lot of progress to full implementation.

There are a few more TODO(PEP612): in the code base but I think they all cover cases where the results are a little looser but should still work acceptably.

(Someone please correct me if I am wrong)

Yeah, I think users should be able to expect PEP 612 to work, and remaining work is better tracked as individual bugs

@hauntsaninja

@hauntsaninja Thanks a lot for working on this feature! Do you happen to know when the new version of mypy which includes it is going to be released?

(Others did much more than I did here) mypy 0.950 should already support most things and was released about a month ago. #12807 for tracking the next release

Navigation Menu

Repository navigation

PEP 612 tracking issue #8645

6 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

PEP 612 tracking issue #8645

Description

Activity

JukkaL commented on Apr 9, 2020

gvanrossum commented on Jul 29, 2020

hauntsaninja commented on Aug 2, 2020

gvanrossum commented on Aug 2, 2020

hauntsaninja commented on Aug 3, 2020

gvanrossum commented on Dec 10, 2020

hauntsaninja commented on Dec 10, 2020

gvanrossum commented on Dec 13, 2020

gvanrossum commented on Jan 2, 2021

sebastian-philipp commented on Aug 25, 2021

graingert commented on Aug 25, 2021

sebastian-philipp commented on Aug 25, 2021

shughes-uk commented on Oct 19, 2021

6 remaining items

erictraut commented on Dec 24, 2021

graingert commented on Dec 24, 2021

NeilGirdhar commented on Dec 24, 2021

kshpytsya commented on Jan 5, 2022

kasium commented on Jan 24, 2022

JelleZijlstra commented on Mar 19, 2022

srittau commented on May 21, 2022

emmatyping commented on May 21, 2022

hauntsaninja commented on May 21, 2022

mristin commented on May 22, 2022

hauntsaninja commented on May 22, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions