Repository navigation
Issues with PyEval_InitThreads and PyGILState_Ensure #70191
Description
| BPO | 26003 |
|---|---|
| Nosy | @pitrou, @vstinner, @tzickel, @iritkatriel |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2021-01-20.11:15:43.800>
created_at = <Date 2016-01-03.19:04:26.863>
labels = ['interpreter-core']
title = 'Issues with PyEval_InitThreads and PyGILState_Ensure'
updated_at = <Date 2021-01-20.11:15:43.799>
user = 'https://github.com/tzickel'bugs.python.org fields:
activity = <Date 2021-01-20.11:15:43.799>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-01-20.11:15:43.800>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2016-01-03.19:04:26.863>
creator = 'tzickel'
dependencies = []
files = ['41485']
hgrepos = []
issue_num = 26003
keywords = []
message_count = 3.0
messages = ['257425', '383400', '385333']
nosy_count = 4.0
nosy_names = ['pitrou', 'vstinner', 'tzickel', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue26003'
versions = ['Python 2.7', 'Python 3.6']
Activity
tzickel commentedon Jan 3, 2016
A few issues regarding threads:
A. (Python 2 & 3) The documentation (https://docs.python.org/3/c-api/init.html) about initializing the GIL/Threading system does not specify that calling PyEval_InitThreads actually binds the calling thread as the main_thread in the ceval.c, meaning that the thread will be in charge till the process goes down for handling Py_AddPendingCall calls, and if it ends/dies, they won't be handled anymore.
This ceval.c's main_thread is different BTW from the one in signalmodule.c which is bound to the thread that called Py_InitializeEx.
Maybe there is sense for both main_thread to be the same one and initialized in the same time ? (even without a GIL existing)
B. (Python 3) Besides the bad documentation regarding this, python 3.4 added issue bpo-19576 which actually hides the call for PyEval_InitThreads inside PyGILState_Ensure. Without careful care and knowledge by the programmer, this might cause for a short lived thread created in C to actually bind the ceval.c's main_thread and when the thread dies main_thread will never be changed again.
The reason this is important is beforehand, the programmer needed to think about PyEval_InitThreads now it's hidden and not even mentioned in the documentation.
C. (Python 2 & 3) In PyEval_InitThreads documentation it's written "It is not safe to call this function when it is unknown which thread (if any) currently has the global interpreter lock." Thus it should be mentioned that PyGILState_Ensure is now calling it in the documentation ?
Also I believe the reason this quote exists is because a potential race condition between thread A which might be running code in PyEval_EvalFrameEx (before PyEval_InitThreads is called, and thus is not GIL aware), and thread B which calls PyEval_InitThreads then calls PyGILState_Ensure, then running Python code, while thread A is still running python code as well. I think it should be explained more clearly in the documentation the implications (race condition).
I think there might be a way to make an PyEval_InitThreads variant which can overcome this race condition. Basically it involves using Py_AddPendingCall to a C function which calls PyEval_InitThreads, and notifies the calling command/thread when it's done. This way we can be sure that the GIL is taken by one thread, and all the others are blocked. (maybe a signal should be sent as well, in case the main_thread is blocked on an I/O operation).
D. (Python 2) If the main_thread finishes it's job, while other python threads are still alive, signal handling isn't processed anymore (Example will be added as a file).
iritkatriel commentedon Dec 19, 2020
Since PyEval_InitThreads is now an empty function and to be removed, this issue seems out of date. A I right?
vstinner commentedon Jan 20, 2021
The GIL is now created by _PyEval_InitGIL() which is called by Py_Initialize(): at Python startup.
_PyRuntime.main_thread is set by _PyRuntime_Initialize() which is also called by Py_Initialize(): at Python startup.
The code to handle signals and pending calls changed a lot since 2016. I close the issue.