PEP 604 NewType

@gvanrossum

BPO	44353
Nosy	@gvanrossum, @ambv, @serhiy-storchaka, @ilevkivskyi, @JelleZijlstra, @pablogsal, @miss-islington, @uriyyo, @wyfo, @domdfcoding, @Fidget-Spinner, @dlax, @AlexWaygood
PRs	bpo-44353: Refactor typing.NewType into callable class #27250 [3.10] bpo-44353: Refactor typing.NewType into callable class (GH-27250) #27258 bpo-44353: Add test to cover __or__ of two typing.NewType #27259 [3.10] bpo-44353: Add test to cover __or__ of two NewType (GH-27259) #27261 bpo-44353: Implement typing.NewType __call__ method in C #27262 bpo-44353: Add tests to cover typing.NewType pickling #27302 bpo-44353: Fix memory leak introduced by #27262 #27305 bpo-44353: Expand NewType tests for complex __qualname__. #27311 bpo-34963: Make the repr of the typing.NewType() result more meaningful. #9951 bpo-44353: Document that typing.NewType is now a class #27319 [3.10] bpo-44353: Document that typing.NewType is now a class (GH-27319) #27321 [3.10] bpo-44353: Expand NewType tests for complex __qualname__ (GH-27311) #27326 [3.10] bpo-44353: Improve tests covering typing.NewType pickling (GH-27302) #27328 bpo-44353: Correct docstring for `NewType` #29785 [3.10] bpo-44353: Correct docstring for `NewType` (GH-29785) #29796
Superseder	bpo-44642: Union of a type and the typing module function

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-07-24.15:29:29.597>
created_at = <Date 2021-06-08.20:21:48.412>
labels = ['3.10', '3.11']
title = 'PEP 604 NewType'
updated_at = <Date 2021-11-26.14:58:16.927>
user = 'https://github.com/wyfo'

bugs.python.org fields:

activity = <Date 2021-11-26.14:58:16.927>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2021-07-24.15:29:29.597>
closer = 'kj'
components = []
creation = <Date 2021-06-08.20:21:48.412>
creator = 'joperez'
dependencies = []
files = []
hgrepos = []
issue_num = 44353
keywords = ['patch']
message_count = 44.0
messages = ['395359', '395512', '395519', '395538', '397299', '397300', '397825', '397826', '397834', '397835', '397838', '397840', '397842', '397847', '397848', '397849', '397869', '397876', '397877', '397878', '397879', '397880', '397881', '397886', '398005', '398006', '398048', '398057', '398080', '398120', '398123', '398129', '398136', '398138', '398139', '398147', '398148', '398149', '398150', '398151', '398152', '398164', '407026', '407055']
nosy_count = 13.0
nosy_names = ['gvanrossum', 'lukasz.langa', 'serhiy.storchaka', 'levkivskyi', 'JelleZijlstra', 'pablogsal', 'miss-islington', 'uriyyo', 'joperez', 'domdfcoding', 'kj', 'dlax', 'AlexWaygood']
pr_nums = ['27250', '27258', '27259', '27261', '27262', '27302', '27305', '27311', '9951', '27319', '27321', '27326', '27328', '29785', '29796']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = '44642'
type = None
url = 'https://bugs.python.org/issue44353'
versions = ['Python 3.10', 'Python 3.11']

typing.NewType doesn't support PEP-604.

>>> import typing
>>> A = typing.NewType("A", int)
>>> B = typing.NewType("B", str)
>>> A | B
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for |: 'function' and 'function'

It is possible to implement NewType to return a class rather than a function as it does currently. However, the class version uses substantially more memory (1072 bytes vs 144 bytes with sys.getsizeof) and NewType is supposed to have almost no runtime overhead.

What about to return callable object instead of function from a typing.NewType?

It will look something like this:

class _NewType:
    __slots__ = ('__name__', '__supertype__')

    def __init__(self, name, supertype):
        self.__name__ = name
        self.__supertype__ = supertype

    def __call__(self, val):
        return val

    def __or__(self, other):
        return Union[self, other]

    def __ror__(self, other):
        return Union[other, self]



def NewType(name, tp):
    """NewType creates simple unique types with almost zero
    runtime overhead. NewType(name, tp) is considered a subtype of tp
    by static type checkers. At runtime, NewType(name, tp) returns
    a dummy callable object that simply returns its argument. Usage::

        UserId = NewType('UserId', int)

        def name_by_id(user_id: UserId) -> str:
            ...

        UserId('user')          # Fails type check

        name_by_id(42)          # Fails type check
        name_by_id(UserId(42))  # OK

        num = UserId(5) + 1     # type: int
    """
    return _NewType(name, tp)

With such implementation __or__ will be available for a NewType and actually because of slots size of object will be 48 bytes (with sys.getsizeof) which is less than current 144 bytes (if memory is important).

python/typing#746 has some previous discussion of implementing NewType as a class (motivated by __repr__), including benchmarks.

Jelle I can try to implement it in C to see if it will give better performance.

@yurii

@yurii, I would like to caution against a C accelerator for the typing module. My reasoning follows:

The maintenance burden is higher. typing is already somewhat complex in Python (lots of MRO/metaclass wizardry). A C module would require knowledge of the C API.
Following from 1., I fear it may raise the barrier for contributions. This is purely anecdotal but I think there are more contributors who know Python than Python + C API.
C API code is much more error-prone than Python code.
It's very hard to find available reviewers for typing-related C changes.
Backports become harder due to point 3. and 4. Also C code needs much more scrutiny. If we cause some obscure bug in Python, it raises an exception; in C it potentially segfaults and kills the interpreter.
Third-party monkey-patching becomes harder.

Unfortunately, I can't offer a good solution to this issue at the moment either.

@ken

@ken thanks for pointing this out.

The only solution which I see is to convert NewType into callable class but it will lead to performance degradation.

@ken what do you think about that?

Do we have to solve it? Using from __future__ import annotations it should work, at least when used as an annotation. Or is that too inconvenient or inconsistent?

Using from __future__ import annotations it should work

Not in type aliases or generic bases. (And if Larry's co_annotations PEP is accepted, it would break in normal annotations too.) I'd prefer to get rid of this sort of subtle difference where the obvious way to write a type only works in some contexts, though I realize that's not possible in all cases.

That means I think we should move forward with making NewTypes into instances with __call__ instead of functions, though according to my benchmarks in the typing issue that will incur a small performance penalty.

I opened PR with refactored typing.NewType into callable class.

Also I have added __module__ attr to typing.NewType so it will become pickable.

Jelle, can you post more details about your benchmark?

New changeset a22b05d by Yurii Karabas in branch 'main':
bpo-44353: Improve tests covering typing.NewType pickling (GH-27302)
a22b05d

New changeset 05f5d8e by Łukasz Langa in branch '3.10':
[3.10] bpo-44353: Expand NewType tests for complex __qualname__ (GH-27311) (GH-27326)
05f5d8e

New changeset e8c0174 by Miss Islington (bot) in branch '3.10':
bpo-44353: Improve tests covering typing.NewType pickling (GH-27302) (GH-27328)
e8c0174

The buildbots are still not green. Unfortunately we will need to revert all PRs if this is not fixed soon as this is already making other issues

@pablo

@pablo, I don't think this change is causing the buildbots to fail. The test failure on all the currently failing buildbots is:

======================================================================
ERROR: test_absolute_circular_submodule (test.test_import.CircularImportTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.installed/build/target/lib/python3.11/test/test_import/__init__.py", line 1355, in test_absolute_circular_submodule
    import test.test_import.data.circular_imports.subpkg2.parent
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'test.test_import.data.circular_imports.subpkg2'

Which seems to be #71486.

Pick any of the failing refleak buildbots. For example:

https://buildbot.python.org/all/#/builders/280/builds/99

You will see:

----------------------------------------------------------------------
Ran 106 tests in 0.074s
OK
......
test_types leaked [2, 2, 2] references, sum=6
test_types leaked [1, 1, 1] memory blocks, sum=3
1 test failed again:
test_types

types is not typing. This is a different failure from what we've seen before. I'll fix that today.

Bisecting points to:

fe13f0b is the first bad commit
commit fe13f0b
Author: Yurii Karabas <[email protected]>
Date: Fri Jul 23 12:47:00 2021 +0300

bpo-44676: Add ability to serialize types.Union (GH-27244)

Lib/test/test_types.py | 35 +++++++++++++++
Lib/typing.py | 4 +-
.../2021-07-19-19-53-46.bpo-44676.WgIMvh.rst | 2 +
Objects/unionobject.c | 51 ++++++++++++++++++++++
4 files changed, 90 insertions(+), 2 deletions(-)
create mode 100644 Misc/NEWS.d/next/Core and Builtins/2021-07-19-19-53-46.bpo-44676.WgIMvh.rst
bisect run success

I am closing this one again, apologies for the confusion.

Thanks a bunch Yurii, Serhiy, Jelle, Łukasz and Pablo for working on this! I'm re-closing this issue. *Fingers-crossed* we won't have to open this again ;-).

New changeset 93c65df by Alex Waygood in branch 'main':
bpo-44353: Correct docstring for NewType (bpo-29785)
93c65df

New changeset 3f024e2 by Miss Islington (bot) in branch '3.10':
bpo-44353: Correct docstring for NewType (GH-29785)
3f024e2

Navigation Menu

Repository navigation

PEP 604 NewType #88519

29 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

PEP 604 NewType #88519

Description

Activity

wyfo commented on Jun 8, 2021

domdfcoding commented on Jun 10, 2021

uriyyo commented on Jun 10, 2021

JelleZijlstra commented on Jun 10, 2021

uriyyo commented on Jul 12, 2021

Fidget-Spinner commented on Jul 12, 2021

uriyyo commented on Jul 19, 2021

gvanrossum commented on Jul 19, 2021

JelleZijlstra commented on Jul 19, 2021

uriyyo commented on Jul 19, 2021

gvanrossum commented on Jul 19, 2021

29 remaining items

ambv commented on Jul 24, 2021

ambv commented on Jul 24, 2021

ambv commented on Jul 24, 2021

pablogsal commented on Jul 24, 2021

Fidget-Spinner commented on Jul 24, 2021

pablogsal commented on Jul 24, 2021

ambv commented on Jul 24, 2021

pablogsal commented on Jul 24, 2021

pablogsal commented on Jul 24, 2021

Fidget-Spinner commented on Jul 24, 2021

gvanrossum commented on Nov 26, 2021

miss-islington commented on Nov 26, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions