Repository navigation
Implemenation of the PEP 492 - Coroutines with async and await syntax #68205
Description
| BPO | 24017 |
|---|---|
| Nosy | @gvanrossum, @ncoghlan, @scoder, @vstinner, @asvetlov, @agronholm, @1st1 |
| Dependencies | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/1st1'
closed_at = <Date 2015-05-13.19:46:15.300>
created_at = <Date 2015-04-20.19:33:55.846>
labels = ['interpreter-core', 'type-feature', 'release-blocker']
title = 'Implemenation of the PEP 492 - Coroutines with async and await syntax'
updated_at = <Date 2015-06-20.19:39:53.655>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2015-06-20.19:39:53.655>
actor = 'yselivanov'
assignee = 'yselivanov'
closed = True
closed_date = <Date 2015-05-13.19:46:15.300>
closer = 'yselivanov'
components = ['Interpreter Core']
creation = <Date 2015-04-20.19:33:55.846>
creator = 'vstinner'
dependencies = ['2292', '22906', '24018', '24180', '24184', '24315', '24316', '24400']
files = ['39148', '39155', '39314', '39325', '39337', '39344', '39354']
hgrepos = []
issue_num = 24017
keywords = ['patch']
message_count = 69.0
messages = ['241678', '241679', '241687', '241697', '242056', '242064', '242086', '242304', '242326', '242627', '242731', '242779', '242793', '242848', '242860', '242862', '242877', '242912', '242927', '242928', '242931', '242935', '242936', '242937', '242938', '242979', '242982', '242984', '243031', '243033', '243035', '243036', '243037', '243038', '243041', '243042', '243117', '243118', '243119', '243759', '244175', '244252', '244257', '244258', '244261', '244264', '244268', '244269', '244273', '244278', '244280', '244281', '244303', '244307', '244319', '244322', '244374', '244384', '244385', '244567', '244568', '244596', '244759', '244761', '244799', '244818', '244826', '244831', '245075']
nosy_count = 8.0
nosy_names = ['gvanrossum', 'ncoghlan', 'scoder', 'vstinner', 'asvetlov', 'alex.gronholm', 'python-dev', 'yselivanov']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24017'
versions = ['Python 3.5']
Activity
1st1 commentedon Apr 20, 2015
Here's the first patch (git diff master..await). Should be easier to review and work from there.
1st1 commentedon Apr 20, 2015
Attaching a patch generated with mercurial
vstinner commentedon Apr 20, 2015
Does the implementation depends on the implementation of the PEP-479? (issue bpo-22906)
Next time, if possible, try to skip generated files. Maybe write a script for that, but sorry I don't know how :-(
1st1 commentedon Apr 21, 2015
Attaching a revised patch (all Victor's comments but asyncio changes)
scoder commentedon Apr 26, 2015
Could we have type slots for the new special methods? Otherwise, implementing the protocol in C would be fairly inefficient, especially for async iteration.
I'm asking because Cython's generator type is not Python's generator type, but implementing the rest of the proposed protocol doesn't seem to be all that difficult.
1st1 commentedon Apr 26, 2015
I don't think it's necessary to have slots for __aiter__, __anext__, __aenter__ and __aexit__. Async iteration will never be as fast as regular iteration, and there is plenty overhead in it. And we don't need slots for async context managers as we don't have slots for regular ones.
What might be a good idea is to add a slot for __await__ method -- tp_await. This will allow to implement Futures in C efficiently. I'd rename 'tp_reserved' for this purpose.
Victor, your thoughts?
1st1 commentedon Apr 26, 2015
In fact I will likely add tp_await in the next PEP iteration. I need it to implement another feature.
scoder commentedon May 1, 2015
You seem to be assuming that the outer loop is the asyncio I/O loop. That might not be the case. It might be a thread-pool executor, it might be an in-memory task switcher, or it might just be something passing on items from a list. At least "__anext__" is worth being fast, IMHO.
Also note that one advantage of slots is that the user can cache their value to keep calling the same C function multiple times. That is not the case for a Python method, which can easily be replaced. Some iterators do that with their __next__ method, and it's perfectly valid. Having to look up the Python method on each iteration and calling through it sounds like unnecessary overhead.
gvanrossum commentedon May 1, 2015
I think we can continue this discussion *after* the PEP's been accepted.
1st1 commentedon May 6, 2015
I'll upload the most recent patch soon.
1st1 commentedon May 7, 2015
Third patch attached. Victor, it would be great if you can review it!
1st1 commentedon May 8, 2015
Another iteration:
45 remaining items
scoder commentedon May 28, 2015
Thanks Yury, I'll give it a try ASAP.
1st1 commentedon May 29, 2015
Stefan,
Because of https://mail.python.org/pipermail/python-committers/2015-May/003410.html I've decided to commit 24315 and 24316 today. Please try to check that everything works before new beta 2.
scoder commentedon May 29, 2015
Tried it, works for me. Thanks!
1st1 commentedon May 29, 2015
This is really good news! Thanks!
python-dev commentedon Jun 1, 2015
New changeset 0708aabefb55 by Yury Selivanov in branch '3.4':
bpo-24017: Fix asyncio.CoroWrapper to support 'async def' coroutines
https://hg.python.org/cpython/rev/0708aabefb55
New changeset 1dc232783012 by Yury Selivanov in branch '3.5':
bpo-24017: Fix asyncio.CoroWrapper to support 'async def' coroutines
https://hg.python.org/cpython/rev/1dc232783012
New changeset 2e7c45560c38 by Yury Selivanov in branch 'default':
bpo-24017: Fix asyncio.CoroWrapper to support 'async def' coroutines
https://hg.python.org/cpython/rev/2e7c45560c38
python-dev commentedon Jun 1, 2015
New changeset a0a699b828e7 by Yury Selivanov in branch '3.5':
bpo-24017: Add a test for CoroWrapper and 'async def' coroutines
https://hg.python.org/cpython/rev/a0a699b828e7
New changeset 89521ac669f0 by Yury Selivanov in branch 'default':
bpo-24017: Add a test for CoroWrapper and 'async def' coroutines
https://hg.python.org/cpython/rev/89521ac669f0
python-dev commentedon Jun 1, 2015
New changeset 1e9e0664ee9b by Yury Selivanov in branch '3.5':
bpo-24017: Make PyEval_(Set|Get)CoroutineWrapper private
https://hg.python.org/cpython/rev/1e9e0664ee9b
New changeset 6fcb64097b1c by Yury Selivanov in branch 'default':
bpo-24017: Make PyEval_(Set|Get)CoroutineWrapper private
https://hg.python.org/cpython/rev/6fcb64097b1c
agronholm commentedon Jun 3, 2015
Was __await__() deliberately left out of concurrent.futures.Future or was that an oversight? Or am I misunderstanding something?
1st1 commentedon Jun 3, 2015
I don't think concurrent.Future is supposed to be used with asyncio (in 'yield from' or 'await' expressions).
scoder commentedon Jun 4, 2015
Hmm, but IMHO a) the new syntax isn't just for asyncio and b) awaiting a Future seems like a *very* reasonable thing to do. I think opening a new ticket for this is a good idea.
1st1 commentedon Jun 4, 2015
Stefan, I honestly have bo idea what concurrent.Future.__await__ would do. There is no loop for concurrent module. If you have a patch with tests in mind, please open a separate issue (targeting 3.6).
gvanrossum commentedon Jun 4, 2015
Maybe it's possible to give an interpretation to awaiting on a threaded
Future? __await__ could return a new asyncio Future, and add a completion
callback to the original Future that makes the asyncio Future ready and
transfers the result/exception. This would have to use
loop.call_soon_threadsafe() to transfer control from the exector thread to
the thread where the loop is running.
The only thing I don't know is whether it's possible for __await__ to
return a Future.
1st1 commentedon Jun 4, 2015
Guido, Stefen, please see bpo-24383.
scoder commentedon Jun 9, 2015
See bpo-24400 regarding a split of yield-based generators and async-def coroutines at a type level.