Higher-Kinded TypeVars

@smheidrich

aka type constructors, generic TypeVars

Has there already been discussion about those?
I do a lot of FP that results in impossible situations because of this. Consider an example:

A = TypeVar('A')
B = TypeVar('B')
F = TypeVar('F')

class M(Generic[F[X], A]):
    def x(fa: F[A], f: Callable[[A], B]) -> F[B]:
        return map(f, fa)

M().x([1], str)

I haven't found a way to make this work, does anyone know a trick or is it impossible?
If not, consider the syntax as a proposal.
Reference implementations would be Haskell, Scala.
optimally, the HK's type param would be indexable as well, allowing for F[X[X, X], X[X]]

Summary of current status (by @smheidrich, 2024-02-08):

@JelleZijlstra has indicated interest in sponsoring a PEP, conditional on a prototype implementation in a major type checker and a well-specified draft PEP.
Drafting the PEP takes place in @nekitdev's fork of the peps repo. The stub PEP draft so far contains a few examples of the proposed syntax.
That same repo's GitHub Discussions forum forum has been designated as the place to discuss the PEP (and presumably the prototype implementation?). Some limited further discussions have taken place there.
- If you want to be notified of new discussion threads, I think you have to set the whole repo as "watched" in GitHub?

I think this came up few times in other discussions, for example one use case is python/mypy#4395. But TBH this is low priority, since such use cases are quite rare.

damn, I searched very thoroughly but did not find this one! 😄
So, consider this my +1!

Adding your +1 is all fine, but who's going to do the (probably pretty complicated) implementation work?

are you approving the feature?

I am neither approving nor disapproving. Just observing that it may be a lot of work for marginal benefits in most codebases.

awfully pragmatic. Where's your sense of adventure? 😄
anyways, I'll work on it, though it's gonna take a while to get into the project.

Similar to microsoft/TypeScript#1213

Not sure if the discussion over there provides any useful insights to the effort over here.

@tek

Hi @tek. I'm also very interested in this, so I'd like to ask if you had any progress with this and volunteer to help if you want.

@rcalsaverini

@rcalsaverini sorry, I've been migrating my legacy code to haskell and am abandoning python altogether. but I wish you great success!

Oh, sad to hear but I see your point. Thanks.

Just to add another use case (which I think relates to this issue):

Using Literal types along with overloading __new__, along with higher-kinded typevars could allow implementing a generic "nullable" ORM Field class, using a descriptor to provide access to the appropriate nullable-or-not field values. The descriptor wouldn't have to be reimplemented in subclasses.

It is one step closer to being possible due to the most recent mypy release's support for honoring the return type of __new__ (python/mypy#1020).

Note: this is basically a stripped-down version of Django's Field class:

# in stub file

from typing import Generic, Optional, TypeVar, Union, overload, Type
from typing_extensions import Literal

_T = TypeVar("_T", bound="Field")
_GT = TypeVar("_GT")

class Field(Generic[_GT]):
    # on the line after the overload: error: Type variable "_T" used with arguments
    @overload
    def __new__(cls: Type[_T], null: Literal[False] = False, *args, **kwargs) -> _T[_GT]: ...
    @overload
    def __new__(cls: Type[_T], null: Literal[True], *args, **kwargs) -> _T[Optional[_GT]]: ...
    def __get__(self, instance, owner) -> _GT: ...

class CharField(Field[str]): ...
class IntegerField(Field[int]): ...
# etc...

# in code

class User:
  f1 = CharField(null=False)
  f2 = CharField(null=True)

reveal_type(User().f1) # Expected: str
reveal_type(User().f2) # Expected: Union[str, None]

I wonder if this is what I need or if there's currently a work around for my (slightly simpler) case?:

I'm building an async redis client with proper type hints. I have a "Commands" class with methods for all redis commands (get, set, exists, strlen ... and hundreds more). Normally each of those methods should return a future (actually coroutine) to the result, but in pipeline mode they should all return None - the commands are added to the pipeline to be executed later.

This is easy enough to implement in python, but not so easy to type hint correctly.

Basic example:

class Redis:
    def execute(self, command) -> Coroutine[Any, Any, Union[None, str, int, float]]:
        return self.connection.execute(...)

    def get(self, *args) -> Coroutine[Any, Any, str]:
        ...
        return self.execute(command)

    def set(self, *args) -> Coroutine[Any, Any, None]:
        ...
        return self.execute(command)

    def exists(self, *args) -> Coroutine[Any, Any, bool]:
        ...
        return self.execute(command)

    # ... and many MANY more ...


class RedisPipeline(Redis):
    def execute(self, command) -> None:
        self.pipeline.append(command)

I tried numerous options to make Coroutine[Any, Any, xxx] generic, but nothing seems to work.

Is there any way around this with python 3.8 and latest mypy? If not a solution would be wonderful - as far as I can think, my only other route for proper types is a script which copy and pastes the entire class and changes the return types in code.

@samuelcolvin

@samuelcolvin I don't think this question belongs in this issue. The reason for the failure (knowing nothing about Redis but going purely by the code you posted) is that in order to make this work, the base class needs to switch to an Optional result, i.e.

    def execute(self, command) -> Optional[Coroutine[Any, Any, Union[None, str, int, float]]]:

I get that, but I need all the public methods to definitely return a coroutine. Otherwise, if it returned an optional coroutine, it would be extremely annoying to use.

What I'm trying to do is modify the return type of many methods on the sub-classes, including "higher kind" types which are parameterised.

Hence thinking it related to this issue.

Not sure why we're talking about Callable semantics here or what it has to do with higher-kinded types

Well, the easy question is, what is the kind of Callable ? This is how I started going down this rabbit hole. This is tightly tied to ParamSpec not being a type either. So in my proposition so far, I decided to introduce :

Types
PackedDomains (the first argument of Callable, so Callable :: PackedDomain -> Type -> Type, but also the kind of ParamSpec)
Domains (basically, for Unpack, which is of kind Type -> Domain)
and domains can furthermore be decomposed into positional domains, keyword domains, and have specific interactions with variadics.

PEP-821 adds support for Unpack[TypedDict] in Callable, and PEP-827 adds a lot of features including more expressive callable types.

Thanks a lot for that, looking into it right now !

This also leads me to question the meaning of expressions like Concatenate[..., P]. More precisely, Concatenate seems to currently accept any number of Ellipsis and ParamSpecs, without any specified semantic. Am I missing an information there, or is it the currently desired behavior ?
More specifically, from what I understand, Concatenate should be kinded as ...[Type] -> PackedDomain -> PackedDomain (with ...[K] representing a variadic number of elements of kind K), and this should forbid those previously questionned uses.

Just a quick progress report, I'm splitting my PEP in 5 PEPs as it was getting really long due to a high number of things being required before properly defining higher kinded typevars. The PEPs will be as follow :
9995 : Defining kinds for actual types (TypeVars would still only represent only concrete types at this level)
9996 : Numerous extensions that naturally comes after defining kinds (like TypedDicts in Concatenate etc)
9997 : Reworking the TypeVar substitution system (defining TypeTemplates etc)
9998 : Adding Higher Kinded TypeVars with only basic features
9999 : Adding advanced higher kinded features (@lift decorator etc)

Again, no ETA, but I can give the current state of things :
9995 : Implementation missing
9996 : Incomplete, lot of things still missing (this is currently what I'm spending most of my time on)
9997 : Specification almost complete, no implementation
9998 : Most of rationale and specification is done, no implementation
9999 : Mostly ideas for now, require more in depth work

I plan on releasing 9995, 9996 and 9997 together at they feel quite connected to me. Then 9998, and finally 9999. Obviously will keep updating here sometimes (feel free to ping me if I don't give updates for too long).

Flesh out DeviceTransferable HKT prototype with concrete Tensor imple…

Flesh out Django Field descriptor HKT prototype

Document container deconstruction in filter_and_count_dups HKT prototype

Flesh out frozendict.__new__ constructor HKT prototype

Clean up Functor HKT prototype and add context header

Clean up Ext/Handler modify HKT prototype and add context header

Simplify generic class factory HKT prototype using Callable

Flesh out MongoModel query prototype and clean up HKT error references

Clean up SettingsRegistry classmethod HKT prototype and compare with …

Flesh out toolz valmap HKT prototype and fix filename typo

Add Trade cross-parameter constructor application HKT prototype

Sorry for the long silence, I've been facing health problems in August and I'm still recovering, so I got really late in regards of where I wish I would have been currently. Obviously, health is my priority so project is on the back burner for now, but it's still in my plans, just extremely slowed down. I'm just really giving a sign of life here.

Navigation Menu

Repository navigation

Higher-Kinded TypeVars #548

212 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Higher-Kinded TypeVars #548

Description

Activity

ilevkivskyi commented on Mar 30, 2018

tek commented on Mar 30, 2018

gvanrossum commented on Mar 31, 2018

tek commented on Mar 31, 2018

gvanrossum commented on Mar 31, 2018

tek commented on Mar 31, 2018

landonpoch commented on Aug 14, 2018

rcalsaverini commented on Jan 10, 2019

tek commented on Jan 10, 2019

rcalsaverini commented on Jan 10, 2019

syastrov commented on Oct 10, 2019

samuelcolvin commented on Apr 26, 2020

gvanrossum commented on Apr 26, 2020

samuelcolvin commented on Apr 26, 2020

212 remaining items

RobinSobczyk commented on May 28, 2026

RobinSobczyk commented on May 28, 2026

RobinSobczyk commented on Jun 19, 2026

RobinSobczyk commented on Sep 22, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions