Skip to content

Access violations in C extension modules on Python 3.9.3 #87876

Closed
Closed
Access violations in C extension modules on Python 3.9.3#87876
Labels
3.9 (EOL)end of lifeOS-windowsextension-modulesC modules in the Modules dirrelease-blockertype-crashA hard crash of the interpreter, possibly with a core dump

Description

@cgohlke
cgohlke
mannequin
opened on Apr 3, 2021
BPO 43710
Nosy @terryjreedy, @gpshead, @pfmoore, @tjguk, @ambv, @markshannon, @zware, @zooba
PRs
  • [3.9] bpo-43710: Change layout of pystate struct to preserve intra-version ABI compatibiliy. #25160
  • [3.9] bpo-43710: Rollback the 3.9 bpo-42500 fix, it broke the ABI in 3.9.3 #25179
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-04-04.11:03:35.837>
created_at = <Date 2021-04-03.03:09:40.678>
labels = ['extension-modules', 'release-blocker', 'OS-windows', 'type-crash', '3.9']
title = 'Access violations in C extension modules on Python 3.9.3'
updated_at = <Date 2021-05-03.21:42:14.320>
user = 'https://bugs.python.org/cgohlke'

    bugs.python.org fields:

    activity = <Date 2021-05-03.21:42:14.320>
actor = 'terry.reedy'
assignee = 'none'
closed = True
closed_date = <Date 2021-04-04.11:03:35.837>
closer = 'lukasz.langa'
components = ['Extension Modules', 'Windows']
creation = <Date 2021-04-03.03:09:40.678>
creator = 'cgohlke'
dependencies = []
files = []
hgrepos = []
issue_num = 43710
keywords = ['patch']
message_count = 5.0
messages = ['390116', '390134', '390172', '390181', '390212']
nosy_count = 9.0
nosy_names = ['terry.reedy', 'gregory.p.smith', 'paul.moore', 'tim.golden', 'cgohlke', 'lukasz.langa', 'Mark.Shannon', 'zach.ware', 'steve.dower']
pr_nums = ['25160', '25179']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue43710'
versions = ['Python 3.9']

    Activity

    1. cgohlke commented on Apr 3, 2021

      @cgohlke
      cgohlkemannequin
      on Apr 3, 2021
      MannequinAuthor

      First reported at numpy/numpy#18720

      After upgrading to Python 3.9.3, 32-bit on Windows, importing numpy (installed via pip install numpy) crashes:

      Microsoft Windows [Version 10.0.19041.906]
C:\Python39-32>python -X dev
Python 3.9.3 (tags/v3.9.3:e723086, Apr  2 2021, 11:01:03) [MSC v.1928 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Windows fatal exception: access violation

Current thread 0x0000105c (most recent call first):
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1116 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1116 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1116 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "C:\Python39-32\lib\site-packages\numpy\random\_pickle.py", line 1 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 790 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1058 in _handle_fromlist
  File "C:\Python39-32\lib\site-packages\numpy\random\__init__.py", line 179 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 790 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1058 in _handle_fromlist
  File "C:\Python39-32\lib\site-packages\numpy\__init__.py", line 156 in <module>
  File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 790 in exec_module
  File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
  File "<stdin>", line 1 in <module>

      Rebuilding numpy from source against Python 3.9.3 fixes this crash, but many other packages compiled against older versions of Python 3.9.x still crash.

      I think this might be due to <8b795ab#diff-7285576bdacf86fe37274d3d0d399c29b4be2959005f60ac0729615d8fca9186%5C%3E.

      In bpo-42500: Fix recursion in or after except (GH-23568) (bpo-24501), the public PyThreadState struct changed. This breaks ABI compatibility for code that is directly accessing the PyThreadState struct members, which is not uncommon. E.g. in Cython https://github.com/cython/cython/blob/0.29.x/Cython/Utility/Exceptions.c#L93-L115. This code will fail on 32-bit. On 64-bit it passes because of fortunate struct alignment AFAICT.

    2. cgohlke
      added
      extension-modulesC modules in the Modules dir
      OS-windows
      3.9 (EOL)end of life
      type-crashA hard crash of the interpreter, possibly with a core dump
      on Apr 3, 2021

    3. ambv commented on Apr 3, 2021

      @ambv
      ambv
      on Apr 3, 2021
      Contributor

      Tomorrow evening CEST (around 30 hours from now) I intend to release a hotfix 3.9.4 with Mark's fix.

    4. ambv
      added
      release-blocker
      on Apr 3, 2021

    5. gpshead commented on Apr 4, 2021

      @gpshead
      gpshead
      on Apr 4, 2021
      Member

      I really suggest a simple rollback (revert) of the change rather than a redo-fix for 3.9.4. Unless you like the idea of a possible 3.9.5 if the redo-fix itself has issues that haven't been given due thought in the rush to undo the 3.9.3 mistake.

      This leaves more time for a proper fix of bpo-42500 in 3.9 not crafted under duress.

    6. ambv commented on Apr 4, 2021

      @ambv
      ambv
      on Apr 4, 2021
      Contributor

      New changeset c7b0fec by Gregory P. Smith in branch '3.9':
      [3.9] bpo-43710: Rollback the 3.9 bpo-42500 fix, it broke the ABI in 3.9.3 (bpo-25179)
      c7b0fec

    7. ambv
      closed this as completedon Apr 4, 2021

    8. 4 remaining items

    9. sai
      removed
      OS-windows
      on May 3, 2021
    10. sai
      assigned
      terryjreedy
      on May 3, 2021
    11. sai
      added
      topic-IDLE
      and removed
      extension-modulesC modules in the Modules dir
      OS-windows
      on May 3, 2021
    12. sai
      assigned
      terryjreedy
      on May 3, 2021
    13. terryjreedy
      added
      extension-modulesC modules in the Modules dir
      OS-windows
      and removed
      topic-IDLE
      on May 3, 2021
    14. terryjreedy
      unassigned
      terryjreedy
      on May 3, 2021
    15. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.9 (EOL)end of lifeOS-windowsextension-modulesC modules in the Modules dirrelease-blockertype-crashA hard crash of the interpreter, possibly with a core dump

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions