You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test_asyncio fails with PYTHONASYNCIODEBUG=1 #76817
vstinner@apu$ PYTHONASYNCIODEBUG=1 ./python -m test test_asyncio -m test_wait_with_exception -v
== CPython 3.7.0a4+ (heads/freebsd_configure:27218edef7, Jan 23 2018, 11:52:08) [GCC 7.2.1 20170915 (Red Hat 7.2.1-2)]
== Linux-4.14.13-300.fc27.x86_64-x86_64-with-fedora-27-Twenty_Seven little-endian
== cwd: /home/vstinner/prog/python/master/build/test_python_24675
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Run tests sequentially
0:00:00 load avg: 1.37 [1/1] test_asyncio
test_wait_with_exception (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... /home/vstinner/prog/python/master/Lib/test/test_asyncio/test_tasks.py:1009: RuntimeWarning: coroutine 'sleep' was never awaited
Coroutine created at (most recent call last)
File "/home/vstinner/prog/python/master/Lib/unittest/case.py", line 615, in run
testMethod()
File "/home/vstinner/prog/python/master/Lib/test/test_asyncio/test_tasks.py", line 1021, in test_wait_with_exception
loop.run_until_complete(self.new_task(loop, foo()))
File "/home/vstinner/prog/python/master/Lib/asyncio/base_events.py", line 423, in run_until_complete
self.run_forever()
File "/home/vstinner/prog/python/master/Lib/asyncio/base_events.py", line 391, in run_forever
self._run_once()
File "/home/vstinner/prog/python/master/Lib/test/test_asyncio/utils.py", line 454, in _run_once
super()._run_once()
File "/home/vstinner/prog/python/master/Lib/asyncio/base_events.py", line 1521, in _run_once
handle._run()
File "/home/vstinner/prog/python/master/Lib/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "/home/vstinner/prog/python/master/Lib/test/test_asyncio/test_tasks.py", line 2289, in _step
return super()._step(*args)
File "/home/vstinner/prog/python/master/Lib/asyncio/coroutines.py", line 61, in send
return self.gen.send(value)
File "/home/vstinner/prog/python/master/Lib/test/test_asyncio/test_tasks.py", line 1009, in sleeper
yield from asyncio.sleep(0.15, loop=loop)
yield from asyncio.sleep(0.15, loop=loop)
Question: there are lots of tests -- for example test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests.test_wait_with_exception, which freezes for me -- that use 'yield from asyncio.sleep(...)', e.g.:
But Andrew switch asyncio.sleep to be an ordinary 'async def' coroutine back in December. How does this work at all?
My tentative hypothesis about the actual bug is that the code above is working in regular mode, but stops working in debug mode, probably because in debug mode @asyncio.coroutine takes a different path involving CoroWrapper and we changed CoroWrapper. But I'm having trouble knowing how to verify or fix that because I don't understand how it ever works in the first place :-)
How confident are we that this is a regression from the coroutine origin tracking changes? (I'd double-check myself, but my cpython checkout is tied up for the next few hours doing --enable-optimizations builds.)
Looking at @asyncio.coroutine, in particular this branch that gets taken when debug mode is enabled:
I'm not seeing anything that would toggle the CO_ITERABLE_COROUTINE flag, and if I use pdb I can see that in the broken test the 'sleeper' function indeed doesn't have that flag set. But I didn't touch that code in the origin tracking patch, which makes me think that the proximal cause here might have been 5f841b5 ?
@asyncio.coroutine has always been buggy: when debug mode is turned on, then it fails to set CO_ITERABLE_COROUTINE.
However, when debug mode is turned on, then traditionally native coroutines got wrapped into CoroWrapper objects. We've always been lazy and reused the same CoroWrapper type for both native coroutines and old-style @asyncio.coroutine coroutines, so CoroWrapper has both coroutine and generator attributes.
This means that until my patch, in debug mode, native coroutines were *getting turned back into generators*. So this hid the bug in @asyncio.coroutine, because it doesn't matter if you can't call native coroutines if you've just turned them into generators.
The fix is trivial, we just need to always use @types.coroutine in @asyncio.coroutine, I'll put up a PR momentarily.
Traceback (most recent call last):
File "/home/vstinner/prog/python/master/Lib/test/test_asyncio/test_tasks.py", line 2473, in test_foobarself.loop.call_later(0.1, fut.set_result(1))
File "/home/vstinner/prog/python/master/Lib/asyncio/base_events.py", line 509, in call_later
context=context)
File "/home/vstinner/prog/python/master/Lib/asyncio/base_events.py", line 522, in call_atself._check_callback(callback, 'call_at')
File "/home/vstinner/prog/python/master/Lib/asyncio/base_events.py", line 556, in _check_callbackf'a callable object was expected by {method}(), 'TypeError: a callable object was expected by call_at(), got None
======================================================================
FAIL: test_nonstream_socket (test.test_asyncio.test_base_events.BaseLoopSendfileTests)
----------------------------------------------------------------------
ValueError: the socket must be non-blocking
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/vstinner/prog/python/master/Lib/test/test_asyncio/test_base_events.py", line 1922, in test_nonstream_socketself.run_loop(self.loop.sock_sendfile(sock, self.file))
AssertionError: "only SOCK_STREAM type" does not match "the socket must be non-blocking"
diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py
index 8d72df6a72..6489f50f27 100644
--- a/Lib/test/test_asyncio/test_base_events.py+++ b/Lib/test/test_asyncio/test_base_events.py@@ -1917,6 +1917,7 @@ class BaseLoopSendfileTests(test_utils.TestCase):
def test_nonstream_socket(self):
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+ sock.setblocking(False)
self.addCleanup(sock.close)
with self.assertRaisesRegex(ValueError, "only SOCK_STREAM type"):
self.run_loop(self.loop.sock_sendfile(sock, self.file))
diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py
index d2162c31b4..9b62d207e4 100644
--- a/Lib/test/test_asyncio/test_tasks.py+++ b/Lib/test/test_asyncio/test_tasks.py@@ -2470,7 +2470,7 @@ class CTask_Future_Tests(test_utils.TestCase):
self.loop = asyncio.new_event_loop()
try:
fut = Fut(loop=self.loop)
- self.loop.call_later(0.1, fut.set_result(1))+ self.loop.call_later(0.1, fut.set_result, 1)
task = asyncio.Task(coro(), loop=self.loop)
res = self.loop.run_until_complete(task)
finally:
Nathaniel: do you want to include these fixes in your PR as well?
Hum, I'm concerned that "self.loop.call_later(0.1, fut.set_result(1))" bug wasn't spotted previously.
I plan to always run the Python test suite (especially when running on our CIs) in the new Python 3.7 "development mode" (python3 -X dev) which enables asyncio debug mode. Are you ok with that?
I plan to always run the Python test suite (especially when running on our CIs) in the new Python 3.7 "development mode" (python3 -X dev) which enables asyncio debug mode. Are you ok with that?
Oh, I forgot to mention the obvious drawback: asyncio debug mode is slower. test_asyncio: 36 seconds => 48 seconds (+12 seconds).
"./python -m test test_asyncio" will not use the asyncio debug mode, only an explicit "./python -X dev -m test test_asyncio" and "make buildbottest".
New changeset fb5a7ad by Yury Selivanov (Nathaniel J. Smith) in branch 'master': bpo-32636: Fix @asyncio.coroutine debug mode bug exposed by #49500 (bpo-5291) fb5a7ad
Activity
vstinner commentedon Jan 23, 2018
(1) test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests.test_wait_with_exception() hangs:
vstinner@apu$ PYTHONASYNCIODEBUG=1 ./python -m test test_asyncio -m test_wait_with_exception -v
== CPython 3.7.0a4+ (heads/freebsd_configure:27218edef7, Jan 23 2018, 11:52:08) [GCC 7.2.1 20170915 (Red Hat 7.2.1-2)]
== Linux-4.14.13-300.fc27.x86_64-x86_64-with-fedora-27-Twenty_Seven little-endian
== cwd: /home/vstinner/prog/python/master/build/test_python_24675
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Run tests sequentially
0:00:00 load avg: 1.37 [1/1] test_asyncio
test_wait_with_exception (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests) ... /home/vstinner/prog/python/master/Lib/test/test_asyncio/test_tasks.py:1009: RuntimeWarning: coroutine 'sleep' was never awaited
Coroutine created at (most recent call last)
File "/home/vstinner/prog/python/master/Lib/unittest/case.py", line 615, in run
testMethod()
File "/home/vstinner/prog/python/master/Lib/test/test_asyncio/test_tasks.py", line 1021, in test_wait_with_exception
loop.run_until_complete(self.new_task(loop, foo()))
File "/home/vstinner/prog/python/master/Lib/asyncio/base_events.py", line 423, in run_until_complete
self.run_forever()
File "/home/vstinner/prog/python/master/Lib/asyncio/base_events.py", line 391, in run_forever
self._run_once()
File "/home/vstinner/prog/python/master/Lib/test/test_asyncio/utils.py", line 454, in _run_once
super()._run_once()
File "/home/vstinner/prog/python/master/Lib/asyncio/base_events.py", line 1521, in _run_once
handle._run()
File "/home/vstinner/prog/python/master/Lib/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "/home/vstinner/prog/python/master/Lib/test/test_asyncio/test_tasks.py", line 2289, in _step
return super()._step(*args)
File "/home/vstinner/prog/python/master/Lib/asyncio/coroutines.py", line 61, in send
return self.gen.send(value)
File "/home/vstinner/prog/python/master/Lib/test/test_asyncio/test_tasks.py", line 1009, in sleeper
yield from asyncio.sleep(0.15, loop=loop)
yield from asyncio.sleep(0.15, loop=loop)
(2) 43 failures and 55 errors:
ERROR: test_context_manager (test.test_asyncio.test_locks.ConditionTests)
ERROR: test_context_manager (test.test_asyncio.test_locks.LockTests)
ERROR: test_context_manager_cant_reuse (test.test_asyncio.test_locks.LockTests)
ERROR: test_lock (test.test_asyncio.test_locks.LockTests)
ERROR: test_lock_by_with_statement (test.test_asyncio.test_locks.LockTests)
ERROR: test_repr (test.test_asyncio.test_locks.LockTests)
ERROR: test_context_manager (test.test_asyncio.test_locks.SemaphoreTests)
ERROR: test_semaphore (test.test_asyncio.test_locks.SemaphoreTests)
ERROR: test_as_completed (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests)
ERROR: test_as_completed_duplicate_coroutines (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests)
ERROR: test_sleep (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests)
ERROR: test_task_cancel_sleeping_task (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests)
ERROR: test_tb_logger_not_called_after_cancel (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests)
ERROR: test_as_completed (test.test_asyncio.test_tasks.CTask_CFuture_SubclassTests)
ERROR: test_as_completed_duplicate_coroutines (test.test_asyncio.test_tasks.CTask_CFuture_SubclassTests)
ERROR: test_sleep (test.test_asyncio.test_tasks.CTask_CFuture_SubclassTests)
ERROR: test_task_cancel_sleeping_task (test.test_asyncio.test_tasks.CTask_CFuture_SubclassTests)
ERROR: test_tb_logger_not_called_after_cancel (test.test_asyncio.test_tasks.CTask_CFuture_SubclassTests)
ERROR: test_as_completed (test.test_asyncio.test_tasks.CTask_CFuture_Tests)
ERROR: test_as_completed_duplicate_coroutines (test.test_asyncio.test_tasks.CTask_CFuture_Tests)
ERROR: test_sleep (test.test_asyncio.test_tasks.CTask_CFuture_Tests)
ERROR: test_task_cancel_sleeping_task (test.test_asyncio.test_tasks.CTask_CFuture_Tests)
ERROR: test_tb_logger_not_called_after_cancel (test.test_asyncio.test_tasks.CTask_CFuture_Tests)
ERROR: test_foobar (test.test_asyncio.test_tasks.CTask_Future_Tests)
ERROR: test_as_completed (test.test_asyncio.test_tasks.CTask_PyFuture_Tests)
ERROR: test_as_completed_duplicate_coroutines (test.test_asyncio.test_tasks.CTask_PyFuture_Tests)
ERROR: test_sleep (test.test_asyncio.test_tasks.CTask_PyFuture_Tests)
ERROR: test_task_cancel_sleeping_task (test.test_asyncio.test_tasks.CTask_PyFuture_Tests)
ERROR: test_tb_logger_not_called_after_cancel (test.test_asyncio.test_tasks.CTask_PyFuture_Tests)
ERROR: test_await_old_style_coro (test.test_asyncio.test_tasks.CompatibilityTests)
ERROR: test_yield_from_awaitable (test.test_asyncio.test_tasks.CompatibilityTests)
ERROR: test_as_completed (test.test_asyncio.test_tasks.PyTask_CFutureSubclass_Tests)
ERROR: test_as_completed_duplicate_coroutines (test.test_asyncio.test_tasks.PyTask_CFutureSubclass_Tests)
ERROR: test_sleep (test.test_asyncio.test_tasks.PyTask_CFutureSubclass_Tests)
ERROR: test_task_cancel_sleeping_task (test.test_asyncio.test_tasks.PyTask_CFutureSubclass_Tests)
ERROR: test_tb_logger_not_called_after_cancel (test.test_asyncio.test_tasks.PyTask_CFutureSubclass_Tests)
ERROR: test_as_completed (test.test_asyncio.test_tasks.PyTask_CFuture_Tests)
ERROR: test_as_completed_duplicate_coroutines (test.test_asyncio.test_tasks.PyTask_CFuture_Tests)
ERROR: test_sleep (test.test_asyncio.test_tasks.PyTask_CFuture_Tests)
ERROR: test_task_cancel_sleeping_task (test.test_asyncio.test_tasks.PyTask_CFuture_Tests)
ERROR: test_tb_logger_not_called_after_cancel (test.test_asyncio.test_tasks.PyTask_CFuture_Tests)
ERROR: test_as_completed (test.test_asyncio.test_tasks.PyTask_PyFuture_SubclassTests)
ERROR: test_as_completed_duplicate_coroutines (test.test_asyncio.test_tasks.PyTask_PyFuture_SubclassTests)
ERROR: test_sleep (test.test_asyncio.test_tasks.PyTask_PyFuture_SubclassTests)
ERROR: test_task_cancel_sleeping_task (test.test_asyncio.test_tasks.PyTask_PyFuture_SubclassTests)
ERROR: test_tb_logger_not_called_after_cancel (test.test_asyncio.test_tasks.PyTask_PyFuture_SubclassTests)
ERROR: test_as_completed (test.test_asyncio.test_tasks.PyTask_PyFuture_Tests)
ERROR: test_as_completed_duplicate_coroutines (test.test_asyncio.test_tasks.PyTask_PyFuture_Tests)
ERROR: test_sleep (test.test_asyncio.test_tasks.PyTask_PyFuture_Tests)
ERROR: test_task_cancel_sleeping_task (test.test_asyncio.test_tasks.PyTask_PyFuture_Tests)
ERROR: test_tb_logger_not_called_after_cancel (test.test_asyncio.test_tasks.PyTask_PyFuture_Tests)
ERROR: test_run_coroutine_threadsafe (test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests)
ERROR: test_run_coroutine_threadsafe_task_cancelled (test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests)
ERROR: test_run_coroutine_threadsafe_with_exception (test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests)
ERROR: test_sleep_zero (test.test_asyncio.test_tasks.SleepTests)
FAIL: test_default_exc_handler_coro (test.test_asyncio.test_base_events.BaseEventLoopTests)
FAIL: test_nonstream_socket (test.test_asyncio.test_base_events.BaseLoopSendfileTests)
FAIL: test_as_completed (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests)
FAIL: test_baseexception_during_cancel (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests)
FAIL: test_baseexception_during_cancel (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests)
FAIL: test_sleep (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests)
FAIL: test_task_cancel_sleeping_task (test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests)
FAIL: test_as_completed (test.test_asyncio.test_tasks.CTask_CFuture_SubclassTests)
FAIL: test_baseexception_during_cancel (test.test_asyncio.test_tasks.CTask_CFuture_SubclassTests)
FAIL: test_baseexception_during_cancel (test.test_asyncio.test_tasks.CTask_CFuture_SubclassTests)
FAIL: test_sleep (test.test_asyncio.test_tasks.CTask_CFuture_SubclassTests)
FAIL: test_task_cancel_sleeping_task (test.test_asyncio.test_tasks.CTask_CFuture_SubclassTests)
FAIL: test_as_completed (test.test_asyncio.test_tasks.CTask_CFuture_Tests)
FAIL: test_baseexception_during_cancel (test.test_asyncio.test_tasks.CTask_CFuture_Tests)
FAIL: test_baseexception_during_cancel (test.test_asyncio.test_tasks.CTask_CFuture_Tests)
FAIL: test_sleep (test.test_asyncio.test_tasks.CTask_CFuture_Tests)
FAIL: test_task_cancel_sleeping_task (test.test_asyncio.test_tasks.CTask_CFuture_Tests)
FAIL: test_as_completed (test.test_asyncio.test_tasks.CTask_PyFuture_Tests)
FAIL: test_baseexception_during_cancel (test.test_asyncio.test_tasks.CTask_PyFuture_Tests)
FAIL: test_baseexception_during_cancel (test.test_asyncio.test_tasks.CTask_PyFuture_Tests)
FAIL: test_sleep (test.test_asyncio.test_tasks.CTask_PyFuture_Tests)
FAIL: test_task_cancel_sleeping_task (test.test_asyncio.test_tasks.CTask_PyFuture_Tests)
FAIL: test_as_completed (test.test_asyncio.test_tasks.PyTask_CFutureSubclass_Tests)
FAIL: test_baseexception_during_cancel (test.test_asyncio.test_tasks.PyTask_CFutureSubclass_Tests)
FAIL: test_baseexception_during_cancel (test.test_asyncio.test_tasks.PyTask_CFutureSubclass_Tests)
FAIL: test_sleep (test.test_asyncio.test_tasks.PyTask_CFutureSubclass_Tests)
FAIL: test_task_cancel_sleeping_task (test.test_asyncio.test_tasks.PyTask_CFutureSubclass_Tests)
FAIL: test_as_completed (test.test_asyncio.test_tasks.PyTask_CFuture_Tests)
FAIL: test_baseexception_during_cancel (test.test_asyncio.test_tasks.PyTask_CFuture_Tests)
FAIL: test_baseexception_during_cancel (test.test_asyncio.test_tasks.PyTask_CFuture_Tests)
FAIL: test_sleep (test.test_asyncio.test_tasks.PyTask_CFuture_Tests)
FAIL: test_task_cancel_sleeping_task (test.test_asyncio.test_tasks.PyTask_CFuture_Tests)
FAIL: test_as_completed (test.test_asyncio.test_tasks.PyTask_PyFuture_SubclassTests)
FAIL: test_baseexception_during_cancel (test.test_asyncio.test_tasks.PyTask_PyFuture_SubclassTests)
FAIL: test_baseexception_during_cancel (test.test_asyncio.test_tasks.PyTask_PyFuture_SubclassTests)
FAIL: test_sleep (test.test_asyncio.test_tasks.PyTask_PyFuture_SubclassTests)
FAIL: test_task_cancel_sleeping_task (test.test_asyncio.test_tasks.PyTask_PyFuture_SubclassTests)
FAIL: test_as_completed (test.test_asyncio.test_tasks.PyTask_PyFuture_Tests)
FAIL: test_baseexception_during_cancel (test.test_asyncio.test_tasks.PyTask_PyFuture_Tests)
FAIL: test_baseexception_during_cancel (test.test_asyncio.test_tasks.PyTask_PyFuture_Tests)
FAIL: test_sleep (test.test_asyncio.test_tasks.PyTask_PyFuture_Tests)
FAIL: test_task_cancel_sleeping_task (test.test_asyncio.test_tasks.PyTask_PyFuture_Tests)
FAIL: test_run_coroutine_threadsafe_task_factory_exception (test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests)
FAILED (failures=43, errors=55, skipped=5)
These errors can be reproduced using attached fails file:
Note: I found these issue when trying to run the full Python test suite using -X dev.
vstinner commentedon Jan 23, 2018
I used this change to skip the test which hangs, (1):
1st1 commentedon Jan 23, 2018
Nathaniel, this is is a regression of fc2f407 -- bpo-32591: Add native coroutine origin tracking.
Please take a look.
njsmith commentedon Jan 23, 2018
Huh, weird. I'll take a look.
njsmith commentedon Jan 24, 2018
Question: there are lots of tests -- for example test.test_asyncio.test_tasks.CTaskSubclass_PyFuture_Tests.test_wait_with_exception, which freezes for me -- that use 'yield from asyncio.sleep(...)', e.g.:
But Andrew switch asyncio.sleep to be an ordinary 'async def' coroutine back in December. How does this work at all?
My tentative hypothesis about the actual bug is that the code above is working in regular mode, but stops working in debug mode, probably because in debug mode @asyncio.coroutine takes a different path involving CoroWrapper and we changed CoroWrapper. But I'm having trouble knowing how to verify or fix that because I don't understand how it ever works in the first place :-)
njsmith commentedon Jan 24, 2018
(However, I can report that at least that particular test starts working again if I convert 'sleeper' into an 'async def' coroutine.)
1st1 commentedon Jan 24, 2018
asyncio.coroutine sets the co_flags bit CO_ITERABLE_COROUTINE for generator functions' code objects.
With that bit flag, Python allows to 'yield from' native coroutines.
CoroWrapper.send() -> wrapped_generator.send() -> YIELD_FROM(native_coro) -> native_coro.send()
njsmith commentedon Jan 24, 2018
How confident are we that this is a regression from the coroutine origin tracking changes? (I'd double-check myself, but my cpython checkout is tied up for the next few hours doing --enable-optimizations builds.)
Looking at @asyncio.coroutine, in particular this branch that gets taken when debug mode is enabled:
cpython/Lib/asyncio/coroutines.py
Lines 135 to 149 in 6b273f7
I'm not seeing anything that would toggle the CO_ITERABLE_COROUTINE flag, and if I use pdb I can see that in the broken test the 'sleeper' function indeed doesn't have that flag set. But I didn't touch that code in the origin tracking patch, which makes me think that the proximal cause here might have been 5f841b5 ?
1st1 commentedon Jan 24, 2018
No, it's this specific commit. I can double check later, but I'm pretty sure about that.
njsmith commentedon Jan 24, 2018
Ah-hah, I get what's going on.
@asyncio.coroutine has always been buggy: when debug mode is turned on, then it fails to set CO_ITERABLE_COROUTINE.
However, when debug mode is turned on, then traditionally native coroutines got wrapped into CoroWrapper objects. We've always been lazy and reused the same CoroWrapper type for both native coroutines and old-style @asyncio.coroutine coroutines, so CoroWrapper has both coroutine and generator attributes.
This means that until my patch, in debug mode, native coroutines were *getting turned back into generators*. So this hid the bug in @asyncio.coroutine, because it doesn't matter if you can't call native coroutines if you've just turned them into generators.
The fix is trivial, we just need to always use @types.coroutine in @asyncio.coroutine, I'll put up a PR momentarily.
vstinner commentedon Jan 24, 2018
Good. Using PR 5291, test_asyncio doesn't hang anymore, and only 2 tests still fail:
======================================================================
ERROR: test_foobar (test.test_asyncio.test_tasks.CTask_Future_Tests)
----------------------------------------------------------------------
======================================================================
FAIL: test_nonstream_socket (test.test_asyncio.test_base_events.BaseLoopSendfileTests)
----------------------------------------------------------------------
ValueError: the socket must be non-blocking
Ran 1874 tests in 49.198s
FAILED (failures=1, errors=1, skipped=5)
vstinner commentedon Jan 24, 2018
Fix for the two remaining bugs:
Nathaniel: do you want to include these fixes in your PR as well?
Hum, I'm concerned that "self.loop.call_later(0.1, fut.set_result(1))" bug wasn't spotted previously.
I plan to always run the Python test suite (especially when running on our CIs) in the new Python 3.7 "development mode" (python3 -X dev) which enables asyncio debug mode. Are you ok with that?
vstinner commentedon Jan 24, 2018
Oh, I forgot to mention the obvious drawback: asyncio debug mode is slower. test_asyncio: 36 seconds => 48 seconds (+12 seconds).
"./python -m test test_asyncio" will not use the asyncio debug mode, only an explicit "./python -X dev -m test test_asyncio" and "make buildbottest".
1st1 commentedon Jan 24, 2018
Victor, can you submit a new PR for this? Let's merge this right away.
vstinner commentedon Jan 24, 2018
Ok, done: #5302
1st1 commentedon Jan 24, 2018
New changeset fb5a7ad by Yury Selivanov (Nathaniel J. Smith) in branch 'master':
bpo-32636: Fix @asyncio.coroutine debug mode bug exposed by #49500 (bpo-5291)
fb5a7ad
1st1 commentedon Jan 24, 2018
New changeset 789e359 by Yury Selivanov (Victor Stinner) in branch 'master':
bpo-32636: Fix two bugs in test_asyncio (bpo-5302)
789e359
1st1 commentedon Jan 24, 2018
This can probably be closed now. Thanks Victor and Nathaniel!