Fix Python embedding across 3.11–3.14 by davidnich · Pull Request #66 · qoretechnologies/module-python

davidnich · 2026-01-20T07:16:37Z

Summary

Fix GIL/TSS handling for Python 3.12–3.14 (stale bound_gilstate, thread-state swapping)
Preserve default sys.path while appending PYTHONHOME/PYTHONPATH entries
Align internals selection for 3.14 and add missing guards
Improve Python 3.11 thread/GIL tracking robustness

Testing

qore test/python.qtest -vv (Python 3.11 build)
qore test/python.qtest -vv (Python 3.12.11 debug build)
qore test/python.qtest -vv (Python 3.13.5 debug build)
qore test/python.qtest -vv (Python 3.14.2 debug build)

Copilot

Pull request overview

This pull request fixes Python embedding support across Python versions 3.11 through 3.14, addressing several critical issues with GIL (Global Interpreter Lock) and TSS (Thread-Specific Storage) handling. The changes ensure proper thread state management, prevent stale thread state references, and improve robustness of Python-Qore integration.

Changes:

Added Python 3.14 internals selection and guard macros to prevent duplicate definitions
Enhanced GIL/TSS handling for Python 3.12-3.14 with bound_gilstate and TSS state consistency checks
Implemented sys.path initialization from PYTHONHOME/PYTHONPATH environment variables for both main and sub-interpreters
Improved Python 3.11 thread state validation with interpreter thread list verification
Added PyDateTimeAPI fallback logic for sub-interpreters where the API may be unavailable
Enhanced import error handling with null checks and proper exception propagation
Added debug logging infrastructure for initialization troubleshooting

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/python314_internals.h	Added QORE_PY_GILSTATE_UNSAFE_DEFINED guard to prevent duplicate _PyGILState_GetInterpreterStateUnsafe definitions
src/python312_internals.h	Added pthread configuration macros, runtime state structures, TSS manipulation functions, and external _PyRuntime declaration
src/python311_internals.h	Enhanced thread state validation in _qore_check_python_created_thread_gil with interpreter thread list checks
src/python-module.h	Simplified Python 3.14 internals selection to use python314_internals.h unconditionally, updated guard conditions
src/python-module.cpp	Added debug logging, sys.path initialization from environment variables, improved bound_gilstate/TSS consistency in GIL helper
src/QorePythonProgram.cpp	Added debug logging, sub-interpreter sys.path setup, PyDateTimeAPI initialization with fallbacks, enhanced thread state validation, improved import error handling

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-20T07:21:44Z

+                size_t start = 0;
+                while (start <= path.size()) {
+                    size_t end = path.find(':', start);
+                    if (end == std::string::npos) {
+                        end = path.size();
+                    }
+                    std::string entry = path.substr(start, end - start);
+                    PyObject* py_entry = PyUnicode_DecodeFSDefault(entry.c_str());
+                    if (py_entry) {
+                        PyList_Append(sys_path, py_entry);
+                        Py_DECREF(py_entry);
+                    }
+                    start = end + 1;
+                }
+            }


The path parsing logic uses a hardcoded colon ':' as the path separator. On Windows, the path separator is semicolon ';'. This code should use the platform-appropriate separator, or handle both separators on all platforms.

Copilot · 2026-01-20T07:21:44Z

+static inline void qore_python_debug_init_log(const char* msg) {
+    const char* debug_init = getenv("QORE_PYTHON_DEBUG_INIT");
+    if (debug_init && *debug_init) {
+        fprintf(stderr, "qore-python init: %s\n", msg);
+        fflush(stderr);
+    }
+}


The debug logging helper function qore_python_debug_init_log is duplicated in both python-module.cpp and QorePythonProgram.cpp. This violates the DRY principle and could lead to maintenance issues. Consider moving this to a shared header file or a common utility source file.

Copilot · 2026-01-20T07:21:45Z

+static inline void qore_python_debug_init_log(const char* msg) {
+    const char* debug_init = getenv("QORE_PYTHON_DEBUG_INIT");
+    if (debug_init && *debug_init) {
+        fprintf(stderr, "qore-python init: %s\n", msg);
+        fflush(stderr);
+    }
+}
+
+static inline void qore_python_debug_init_log_symbol(const char* symbol) {
+    const char* debug_init = getenv("QORE_PYTHON_DEBUG_INIT");
+    if (debug_init && *debug_init) {
+        fprintf(stderr, "qore-python init: importing symbol '%s'\n", symbol ? symbol : "<null>");
+        fflush(stderr);
+    }


The debug logging helper functions qore_python_debug_init_log and qore_python_debug_init_log_symbol are duplicated from python-module.cpp. This violates the DRY principle and could lead to maintenance issues. Consider moving these to a shared header file or a common utility source file.

Suggested change

static inline void qore_python_debug_init_log(const char* msg) {

const char* debug_init = getenv("QORE_PYTHON_DEBUG_INIT");

if (debug_init && *debug_init) {

fprintf(stderr, "qore-python init: %s\n", msg);

fflush(stderr);

}

}

static inline void qore_python_debug_init_log_symbol(const char* symbol) {

const char* debug_init = getenv("QORE_PYTHON_DEBUG_INIT");

if (debug_init && *debug_init) {

fprintf(stderr, "qore-python init: importing symbol '%s'\n", symbol ? symbol : "<null>");

fflush(stderr);

}

static inline bool qore_python_debug_init_enabled() {

const char* debug_init = getenv("QORE_PYTHON_DEBUG_INIT");

return debug_init && *debug_init;

}

static inline void qore_python_debug_init_log(const char* msg) {

if (!qore_python_debug_init_enabled()) {

return;

}

fprintf(stderr, "qore-python init: %s\n", msg);

fflush(stderr);

}

static inline void qore_python_debug_init_log_symbol(const char* symbol) {

if (!qore_python_debug_init_enabled()) {

return;

}

fprintf(stderr, "qore-python init: importing symbol '%s'\n", symbol ? symbol : "<null>");

fflush(stderr);

Copilot · 2026-01-20T07:21:45Z

+        }
+        if (sys_path && PyList_Check(sys_path)) {
+            size_t start = 0;
+            while (start <= path.size()) {


The loop condition while (start <= path.size()) will process one extra iteration when start == path.size(), which will create an empty substring. This happens after processing a path that ends with a colon, or at the very end of the path. The condition should be while (start < path.size()) to avoid adding an empty string to sys.path.

Suggested change

while (start <= path.size()) {

while (start < path.size()) {

Copilot · 2026-01-20T07:21:45Z

+            }
+            if (sys_path && PyList_Check(sys_path)) {
+                size_t start = 0;
+                while (start <= path.size()) {


The loop condition while (start <= path.size()) will process one extra iteration when start == path.size(), which will create an empty substring. This happens after processing a path that ends with a colon, or at the very end of the path. The condition should be while (start < path.size()) to avoid adding an empty string to sys.path.

Suggested change

while (start <= path.size()) {

while (start < path.size()) {

Copilot · 2026-01-20T07:21:45Z

+            size_t start = 0;
+            while (start <= path.size()) {
+                size_t end = path.find(':', start);
+                if (end == std::string::npos) {
+                    end = path.size();
+                }
+                std::string entry = path.substr(start, end - start);
+                PyObject* py_entry = PyUnicode_DecodeFSDefault(entry.c_str());
+                if (py_entry) {
+                    PyList_Append(sys_path, py_entry);
+                    Py_DECREF(py_entry);
+                }
+                start = end + 1;
+            }


The path parsing logic uses a hardcoded colon ':' as the path separator. On Windows, the path separator is semicolon ';'. This code should use the platform-appropriate separator, or handle both separators on all platforms.

Fix Python 3.11-3.14 embedding and GIL handling

b85fd44

Copilot AI review requested due to automatic review settings January 20, 2026 07:16

Copilot started reviewing on behalf of davidnich January 20, 2026 07:17 View session

Copilot AI reviewed Jan 20, 2026

View reviewed changes

Address PR review feedback

3a602f6

davidnich merged commit e691183 into develop Jan 20, 2026
1 check passed

davidnich deleted the bugfix/python_fixes branch January 20, 2026 07:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix Python embedding across 3.11–3.14#66

Fix Python embedding across 3.11–3.14#66
davidnich merged 2 commits into
developfrom
bugfix/python_fixes

davidnich commented Jan 20, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 20, 2026

Uh oh!

Copilot AI Jan 20, 2026

Uh oh!

Copilot AI Jan 20, 2026

Uh oh!

Copilot AI Jan 20, 2026

Uh oh!

Copilot AI Jan 20, 2026

Uh oh!

Copilot AI Jan 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

davidnich commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

davidnich commented Jan 20, 2026 •

edited

Loading