Concurrent imports of a package and its submodule raise _DeadlockError in 3.14 and since 3.13.16 #160045

Bug description:

Since the fix for gh-130094 (GH-130101, backported to 3.13 in GH-155915 and first released in 3.13.16), _find_and_load_unlocked() re-imports a parent package whose spec is still _initializing, and it does so while holding the child module's lock. That creates a lock-order inversion between two common imports that worked before:

• Thread A runs import pkg. It holds pkg's module lock while pkg/__init__.py runs, and that file does from .sub import X, so A then needs the lock for pkg.sub.
• Thread B runs import pkg.sub while pkg is still initializing. It takes the pkg.sub lock, sees pkg.__spec__._initializing, and now waits for pkg's lock.

importlib detects the cycle and raises _DeadlockError in thread A. Up to 3.13.15, thread B loaded pkg.sub without waiting for the parent, so both imports succeeded. There is no import cycle in user code, and the same imports work in either order on a single thread.

Reproducer (fails every time on 3.13.16 and 3.14.4; passes on 3.13.13):

import os, sys, tempfile, threading, time

root = tempfile.mkdtemp()
os.makedirs(os.path.join(root, "pkg"))
with open(os.path.join(root, "pkg", "__init__.py"), "w") as f:
    f.write("import time\ntime.sleep(0.5)\nfrom .sub import X\n")
with open(os.path.join(root, "pkg", "sub.py"), "w") as f:
    f.write("X = 1\n")
sys.path.insert(0, root)

errors = []

def run(name):
    try:
        __import__(name)
    except BaseException as e:
        errors.append(f"{name}: {type(e).__name__}: {e}")

a = threading.Thread(target=run, args=("pkg",))
b = threading.Thread(target=run, args=("pkg.sub",))
a.start()
time.sleep(0.1)  # pkg is now initializing; thread a holds its module lock
b.start()
a.join(); b.join()
print(sys.version.split()[0], "->", errors or "OK")

3.13.13 -> OK
3.13.16 -> ["pkg: _DeadlockError: deadlock detected by _ModuleLock('pkg.sub') at 0x..."]
3.14.4  -> ["pkg: _DeadlockError: deadlock detected by _ModuleLock('pkg.sub') at 0x..."]

Removing only the new block below from 3.13.16's _find_and_load_unlocked() makes the reproducer pass. The meta_path = list(meta_path) part of the same change has no effect on it:

        if getattr(parent_spec, '_initializing', False):
            _call_with_frames_removed(import_, parent)
        # Crazy side-effects (again)!
        module = sys.modules.get(name)
        if module is not None:
            return module

Real-world impact: any package whose __init__ imports its own submodules is affected (requests, prometheus_client, and so on) as soon as two threads import it and one of its submodules for the first time at once. The Datadog Agent loads Python checks from several threads and hit this right after moving to 3.13.16. Checks failed at startup with _DeadlockError: deadlock detected by _ModuleLock('requests.exceptions'), so we went back to 3.13.15 (DataDog/datadog-agent#57902). Other projects report the same error after upgrading to 3.13.16 or 3.14, for example gke-labs/kube-agents#2446, dgunning/edgartools#1325 and marimo-team/marimo#11095.

Because this is a regression in a bugfix release, could the 3.13 backport be reverted, or this part changed so the thread doesn't wait for the parent's lock while holding the child's? The same code is on 3.14 and main.

gh-159963 is related but separate: it covers a 3.15-only change from gh-83065. This report is about the gh-130094 change that is already in 3.13.16 and 3.14.

CPython versions tested on:

3.13, 3.14

Operating systems tested on:

macOS