This is an addendum to issue35621:

To be able to call `asyncio.create_subprocess_exec()` from another thread, A separate event loop needs to be created. To make the child watcher aware of this new loop, I have to call `asyncio.get_child_watcher().attach_loop(loop)`. However, in the current implementation this call needs to be made by the main thread (or else the `signal` module will complain as handlers may only be registered in the main thread).

So, to work around the above limitations, the following workflow needs to be used:

1) create a new loop in the main thread
2) attach it to the child watcher
3) spawn a worker thread
4) set the previously created event loop as default loop

After that, I can run `asyncio.create_subprocess_exec()` in the worker thread. However, I suppose the worker thread will be the only thread able to call that function, given the child watcher's limitation to a single loop.

Am I missing something ? Given the complexity of this, I would expect this to be better documented in the sections explaining how `asyncio.subprocess` and `threading` interact.

> Am I missing something ? Given the complexity of this, I would expect this to be better documented in the sections explaining how `asyncio.subprocess` and `threading` interact.

The current documentation says:

"To handle signals and to execute subprocesses, the event loop must be run in the main thread."

https://docs.python.org/dev/library/asyncio-dev.html#concurrency-and-multithreading

But I agree that the doc can be enhanced :-) Do you have suggestions?

By the way, asyncio doc is outdated:
"The default asyncio event loop implementation on Windows does not support subprocesses. Subprocesses are available for Windows if a ProactorEventLoop is used. See Subprocess Support on Windows for details."
https://docs.python.org/dev/library/asyncio-subprocess.html#creating-subprocesses

It's no longer true in Python 3.8:

"Changed in version 3.8: On Windows, ProactorEventLoop is now the default event loop."
https://docs.python.org/dev/library/asyncio-platforms.html

That's quite an unfortunate limitation ! I'm working on a GUI frontend to a Python tool I wrote using asyncio, and the GUI (Qt-based) itself insists to run its own event loop in the main thread.

I'm not sure how to work around this limitation, but I can report that my previously reported strategy appears to be working well (so far).

What are the issues I should expect to encounter running an asyncio event loop in a worker thread ?

OK, so while I have been able to work around the issues (by using `quamash` to bridge between `asyncio` and `Qt`), I'd still like to understand the rationale behind the limitation that any subprocess-managing event-loop has to run in the main thread. Is this really an architectural limitation or a limit of the current implementation ?

And to your question: As I wasn't really expecting such a limitation, I would have expected 
    "To handle signals and to execute subprocesses, the event loop must be run in the main thread."

to be written much more prominently (as a warning admonition even, perhaps).

The limitation is a consequence of how Linux works.
Unix has no cross-platform API for non-blocking waiting for child process finish except handling SIGCHILD signal.

On the other hand signal handlers in Python should work in the main thread.

Your trick with a loop creation in the main thread and actual running in another thread can work, but asyncio doesn't guarantee it.
The behavior can be broken in next releases, sorry.

IIRC we discussed this scenario recently, official support for transferring a loop between threads is a too strict limitation, not all third-party implementations are ready for that.

> The limitation is a consequence of how Linux works.
> Unix has no cross-platform API for non-blocking waiting for child process finish except handling SIGCHILD signal.

Why does the `wait()` have to be non-blocking ? We can call it once in 
response to the reception of a `SIGCHILD`, where we know the call 
wouldn't block. Then we can pass the `pid` to whatever event loop 
created the subprocess to do the cleanup there...

> On the other hand signal handlers in Python should work in the main thread.

That's fine.

> Your trick with a loop creation in the main thread and actual running in another thread can work, but asyncio doesn't guarantee it.
> The behavior can be broken in next releases, sorry.

Yeah, I observed some strange issues that looked like they could be 
fixed by someone intimately familiar with `asyncio`. But given the 
documented limitation, it seemed wise not to descend into that rabbit 
hole, and so I (at least temporarily) abandoned the entire approach.

--

Stefan

       ...ich hab' noch einen Koffer in Berlin...

Fixed by #35621