PEP 572: Assignment Expressions

@gvanrossum

BPO	35224
Nosy	@gvanrossum, @tim-one, @warsaw, @rhettinger, @ncoghlan, @ericvsmith, @nedbat, @stevendaprano, @serhiy-storchaka, @koobs, @matrixise, @willingc, @vedgar, @emilyemorehouse, @joernheissler, @pablogsal, @miss-islington, @tirkarthi
PRs	bpo-35224: PEP 572 Implementation #10497 bpo-35224: PEP 572 Implementation #10497 bpo-35224: Add support for NamedExpr to unparse.py #11670 bpo-35224: Add support for NamedExpr to unparse.py #11670 bpo-35224: Add support for NamedExpr to unparse.py #11670 bpo-35224: Add PEP 572 (assignment expressions) to What's New #12941 Non-working attempt to introduce := as an assignment operator #6333 bpo-35224: Reverse evaluation order of key: value in dict comprehensions #14139 bpo-35224: Bump the pyc magic number after the change in MAP_ADD #14313 [3.8] bpo-35224: Reverse evaluation order of key: value in dict comprehensions (GH-14139) #14314 [3.8] bpo-35224: Bump the pyc magic number after the change in MAP_ADD (GH-14313) #14315 bpo-35224: Add What's new entry for evaluation order in dict comprehensions #14319 bpo-35224: Bump the pyc magic number by 1 instead of by 10 in last modification #14320 [3.8] bpo-35224: Bump the pyc magic number by 1 instead of by 10 in last modification (GH-14320) #14321 [3.8] bpo-35224: Add What's new entry for evaluation order in dict comprehensions (GH-14319) #14361 bpo-35224: Additional documentation for Assignment Expressions #15935 [3.8] bpo-35224: Additional documentation for Assignment Expressions (GH-15935) #15967

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/emilyemorehouse'
closed_at = <Date 2019-09-11.15:16:49.055>
created_at = <Date 2018-11-13.00:52:17.907>
labels = ['interpreter-core', 'type-feature', '3.8']
title = 'PEP 572: Assignment Expressions'
updated_at = <Date 2019-09-11.15:29:33.173>
user = 'https://github.com/emilyemorehouse'

bugs.python.org fields:

activity = <Date 2019-09-11.15:29:33.173>
actor = 'gvanrossum'
assignee = 'emilyemorehouse'
closed = True
closed_date = <Date 2019-09-11.15:16:49.055>
closer = 'emilyemorehouse'
components = ['Interpreter Core']
creation = <Date 2018-11-13.00:52:17.907>
creator = 'emilyemorehouse'
dependencies = []
files = []
hgrepos = []
issue_num = 35224
keywords = ['patch', 'patch']
message_count = 64.0
messages = ['329781', '334325', '334328', '334330', '334331', '334332', '334334', '334341', '334358', '334839', '334840', '335437', '335438', '335452', '335458', '339098', '339112', '339121', '339131', '339159', '339160', '339172', '340770', '340774', '340776', '340777', '340792', '340794', '345782', '345791', '345798', '346282', '346287', '346290', '346291', '346292', '346298', '346303', '346304', '346305', '346306', '346307', '346308', '346312', '346319', '346327', '346329', '346331', '346472', '346473', '347985', '349011', '349013', '349017', '349019', '349021', '349024', '349025', '349026', '349034', '351906', '351929', '351933', '351940']
nosy_count = 18.0
nosy_names = ['gvanrossum', 'tim.peters', 'barry', 'rhettinger', 'ncoghlan', 'eric.smith', 'nedbat', 'steven.daprano', 'serhiy.storchaka', 'koobs', 'matrixise', 'willingc', 'veky', 'emilyemorehouse', 'joernheissler', 'pablogsal', 'miss-islington', 'xtreak']
pr_nums = ['10497', '10497', '11670', '11670', '11670', '12941', '6333', '14139', '14313', '14314', '14315', '14319', '14320', '14321', '14361', '15935', '15967']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue35224'
versions = ['Python 3.8']

This issue will serve to track development and PRs for the implementation of PEP-572: Assignment Expressions.

New changeset 8f59ee0 by Emily Morehouse in branch 'master':
bpo-35224: PEP-572 Implementation (bpo-10497)
8f59ee0

This is huge!

I do recall there are some minor edge cases where the implementation currently doesn't match the PEP. Could you summarize those here, and add your recommendation (e.g. change the PEP, fix the code, wait and see) with motivation?

The change broke most buildbots: congrats Emily, each core dev has to do their as part of their training ;-) Don't worry, it's fine.

I wrote PR bpo-11670 which should fix test_tools.

@vstinner

@vstinner Is there something I could/should have checked other than the CI displayed in GitHub before merging? Let me know if I can help.

Here's a brief summary of the differences between the PEP spec and implementation:

From the "Scope of the target" section of the PEP, there are two cases that should raise a TargetScopeError: when an assignment expression is used in a comprehension inside a class body or for special cases in comprehensions.

Invalid examples for the latter include:

[i := i+1 for i in range(5)]
[[(j := j) for i in range(5)] for j in range(5)]
[i := 0 for i, j in stuff]
[i+1 for i in i := stuff]

However, the following work in the implementation,though the PEP states they should be invalid:

    >>> [i := i+1 for i in range(5)]
    [1, 2, 3, 4, 5]
    >>> i
    5

    >>> [i := 0 for i, j in [(1, 2)]]
    [0]

The following does not work in the implementation (as desired), but does not throw a TargetScopeError as defined in the PEP:

    >>> [i+1 for i in i := range(5)]
    File "<stdin>", line 1
        [i+1 for i in i := range(5)]
                        ^
    SyntaxError: invalid syntax

IMO, I was leaning towards advocating for changing the PEP to match the implementation. I think the error messages are clear and expected, and restricting what already works would require significant special cases. I'm open to discussion though.

There's also documentation that should certainly be added (and I believe a spot where assignment expressions are explicitly mentioned as not being included in the language, which is no longer the case)

@vstinner

@vstinner Is there something I could/should have checked other than the CI displayed in GitHub before merging? Let me know if I can help.

It wasn't your fault. Our pre-commit checks on pull requests is incomplete on purpose: it has to be fast. It's fine to break buildbots sometimes. It's a tradeoff.

If you want to help, please merge #11670 as soon as the CI test pass since I'm going to bed :-)

You are the victim of a very very specific annoying test, test_unparse with its annoying "randomly pick 10 files from the stdlib" feature...

New changeset 1396d8f by Victor Stinner in branch 'master':
bpo-35224: Add support for NamedExpr to unparse.py (GH-11670)
1396d8f

I don't know if this is the correct issue for questions/clarifications but it seems parens are mandatory while using named expressions in while statement which makes some of the examples invalid like https://www.python.org/dev/peps/pep-0572/#sysconfig-py . From my limited knowledge while statement Grammar was not modified at https://github.com/python/cpython/pull/10497/files#diff-cb0b9d6312c0d67f6d4aa1966766ceddR73 and no tests for while statement which made me assume it's intentional. I haven't followed the full discussion about PEP-572 so feel free to correct me if it's a conscious decision and in that case the PEP-572 can be updated.

# python info

➜ cpython git:(master) ./python.exe
Python 3.8.0a0 (heads/bpo35113-dirty:49329a217e, Jan 25 2019, 09:57:53)
[Clang 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>

# Example as in PEP-572 to create a simple file that reads itself and prints lines that matches "foo"

➜ cpython git:(master) cat /tmp/foo.py
import re

with open("/tmp/foo.py") as f:
    while line := f.readline():
        if match := re.search(r"foo", line):
            print(match.string.strip("\n"))
➜  cpython git:(master) ./python.exe /tmp/foo.py
  File "/tmp/foo.py", line 4
    while line := f.readline():
               ^
SyntaxError: invalid syntax

# Wrapping named expression with parens for while makes this valid

➜ cpython git:(master) cat /tmp/foo.py
import re

with open("/tmp/foo.py") as f:
    while (line := f.readline()):
        if match := re.search(r"foo", line):
            print(match.string.strip("\n"))
➜  cpython git:(master) ./python.exe /tmp/foo.py
with open("/tmp/foo.py") as f:
        if match := re.search(r"foo", line):

As a user I think parens shouldn't be mandatory in while statement since if statement works fine. Parens can cause while statement to be superfluous in some cases and an extra case to remember while teaching.

Note: I checked and 3.x buildbots are back to green (ignoring the ones which already failed previously). Good.

FYI, we need a prominent Whatsnew entry for this.

@rhettinger

@rhettinger absolutely, I'm going to include that in my documentation PR which is currently in progress. :)

Did the documentation PR get pushed/merged? Emily mentioned having one in progress above, but it doesn't appear in the linked PRs.

https://bugs.python.org/issue37757 separates out the TargetScopeError handling for conflicts between assignment expressions and comprehension iteration variables.

Also, a major procedural note: it is NOT OK to merge a PEP implementation that completely ignores parts of the PEP. The merged tests are actively forcing NON-compliance with the accepted PEP, since they're requiring implementations to accept code that the PEP explicitly states should be disallowed.

Those rules were added because the behaviour in CPython leaks CPython implementation details that we *don't want* to be part of the language specification.

FWIW, I'm working on an improved whatsnew entry in #15127

Thanks for catching that this was still incomplete.

Also, a major procedural note: it is NOT OK to merge a PEP implementation that completely ignores parts of the PEP. The merged tests are actively forcing NON-compliance with the accepted PEP, since they're requiring implementations to accept code that the PEP explicitly states should be disallowed.

It was known the implementation was unfinished in this respect, but it was deemed better to merge what we had lest the work be lost in merge conflicts, and iterate in later betas. I've written some code that uses the walrus operator and have found it quite solid. The early existence of an implementation (albeit incomplete) has also helped get support for this in mypy (python/mypy#6899).

I don't recall being aware that there were tests that specifically *checked* that the implementation was incomplete, and that's obviously wrong.

bpo-37757 now has an associated PR adding the missing TargetScopeError cases: #15131

There's one case where it goes beyond what the PEP specifies: because the outermost iterable expression gets evaluated in a different scope from the rest of the comprehension, it just flat out prohibits the use of assignment expressions in comprehension iterable expressions.

This was one of the cases where we explicitly didn't want the CPython implementation behaviour to leak into the language specification (as name binding in the outermost iterable expression would create an unrelated binding in the containing scope, while name binding in other iterable expressions would rebind any conflicting iteration variable in the comprehension), so the current PR takes the more conservative path, and defers allowing name binding in the iterable expressions until a specific use case for doing so is presented).

Thanks Guido. The former test cases that the new PR removes are the following:

res = [i := i for i in range(5)]
res = [i := 0 for i, j in [(1, 2), (3, 4)]]
res = [(i := 0, j := 1) for i, j in [(1, 2), (3, 4)]]
res = [(i := i, j := j) for i, j in [(1, 2), (3, 4)]]
res = [(i := j, j := i) for i, j in [(1, 2), (3, 4)]]

These all raise TargetScopeError with the PR applied:

>>> res = [i := i for i in range(5)]
  File "<stdin>", line 1
TargetScopeError: named expression cannot rebind comprehension iteration variable
>>> res = [i := 0 for i, j in [(1, 2), (3, 4)]]
  File "<stdin>", line 1
TargetScopeError: named expression cannot rebind comprehension iteration variable
>>> res = [(i := 0, j := 1) for i, j in [(1, 2), (3, 4)]]
  File "<stdin>", line 1
TargetScopeError: named expression cannot rebind comprehension iteration variable
>>> res = [(i := i, j := j) for i, j in [(1, 2), (3, 4)]]
  File "<stdin>", line 1
TargetScopeError: named expression cannot rebind comprehension iteration variable
>>> res = [(i := j, j := i) for i, j in [(1, 2), (3, 4)]]
  File "<stdin>", line 1
TargetScopeError: named expression cannot rebind comprehension iteration variable

Can you suggest a PEP update too, for the case that goes beyond the PEP?
And please provide examples (not everybody knows immediately what
"outermost iterable expression" refers to. :-)

Proposed PEP update is here: python/peps#1140

The update also aims to clarify *why* we're doing the extra work in CPython's compiler to make these cases fail (i.e. we don't want to implicitly impose the current CPython runtime behaviour on other implementations)

New changeset 6357c95 by Miss Islington (bot) (Emily Morehouse) in branch 'master':
bpo-35224: Additional documentation for Assignment Expressions (GH-15935)
6357c95

New changeset be2aa58 by Stéphane Wirtel (Miss Islington (bot)) in branch '3.8':
bpo-35224: Additional documentation for Assignment Expressions (GH-15935) (GH-15967)
be2aa58

All areas that were identified for additional work have been addressed.

If there is anything else that needs to be improved or updated, please create a new issue.

Thanks!

Congrats! Let's party.

Navigation Menu

PEP 572: Assignment Expressions #79405

40 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

PEP 572: Assignment Expressions #79405

Description

Activity

emilyemorehouse commented on Nov 13, 2018

emilyemorehouse commented on Jan 24, 2019

gvanrossum commented on Jan 25, 2019

vstinner commented on Jan 25, 2019

emilyemorehouse commented on Jan 25, 2019

vstinner commented on Jan 25, 2019

vstinner commented on Jan 25, 2019

tirkarthi commented on Jan 25, 2019

vstinner commented on Jan 25, 2019

rhettinger commented on Feb 4, 2019

emilyemorehouse commented on Feb 4, 2019

40 remaining items

ncoghlan commented on Aug 5, 2019

ncoghlan commented on Aug 5, 2019

ncoghlan commented on Aug 5, 2019

rhettinger commented on Aug 5, 2019

gvanrossum commented on Aug 5, 2019

ncoghlan commented on Aug 5, 2019

ncoghlan commented on Aug 5, 2019

gvanrossum commented on Aug 5, 2019

ncoghlan commented on Aug 5, 2019

miss-islington commented on Sep 11, 2019

matrixise commented on Sep 11, 2019

emilyemorehouse commented on Sep 11, 2019

gvanrossum commented on Sep 11, 2019

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions