Skip to content

Python.exe crashes if __pycache__ directories exist (garbage collection trashed) #481

New issue
New issue
Closed
Closed
Python.exe crashes if __pycache__ directories exist (garbage collection trashed)#481
Labels
bug
Milestone
2.5.0

Description

@testrunner123
testrunner123
opened on May 31, 2017

Environment

  • Pythonnet version: 2.3.0
  • Python version: 3.5 64-bit
  • Operating System: Windows 7 Pro 64-bit SP1

Details

  • Describe what you were trying to get done.

I am trying to import .NET assemblies and use them together. Unfortunately, python interpreter always crashes if .py files are already precompiled and lay inside pycache directories. I tried breifly python 3.6 and newer pythonnet, but it still crashes. However it doesn't crash with old python 2.7.13 64-bit (need to recompile pythonnet for 2.7).

I used Debugger and see that python garbage collection is trashed. Mostly object->tp_is_gc field is invalid not showing to real function.

Deleteing pycache directories allow test.py to run once.

Please, help in identify the issue and in making the fix.

  • What commands did you run to trigger this issue?

pythonGCCrash.zip

  1. Unpack zip, open Visual Studio 2015 Update 3 Command Prompt in this directory
  2. Need to compile pythonnet for python 3.5 64-bit windows. Copy clr.pyd and Python.Runtime.dll to the directory of run.bat
  3. start run.bat
  4. python.exe crashes after printing "0"
  • If there was a crash, please include the traceback here.
    The crash happens mostly inside visit_decref() function in PyObject_IS_GC(op) macro. It shows either "op" is 0 or "op->tp_is_gc" has a wrong address.

Activity

  1. den-run-ai commented on May 31, 2017

    @den-run-ai
    den-run-ai
    on May 31, 2017
    Contributor

    did you try running python with -B flag?

  2. testrunner123 commented on Jun 1, 2017

    @testrunner123
    testrunner123
    on Jun 1, 2017
    ContributorAuthor

    did you try running python with -B flag?

    I've tried this: in 4 of 5 attempts, python interpreter 3.5 64-bit crashes at the exit. No pycache directores exist in this case.

  3. vmuriart commented on Jun 1, 2017

    @vmuriart
    vmuriart
    on Jun 1, 2017
    Contributor

    I haven't seen yoyr sample code, this isn't related to the sub classing bug right? I'm curious if pycache existed when the subclass tests failed

  4. den-run-ai commented on Jun 1, 2017

    @den-run-ai
    den-run-ai
    on Jun 1, 2017
    Contributor

    You zip file does not include Axa.* .NET dependencies.

  5. testrunner123 commented on Jun 1, 2017

    @testrunner123
    testrunner123
    on Jun 1, 2017
    ContributorAuthor

    @vmuriart
    I do subclass in sample and I believe it is related. In real assembly I have subclassing and strange effects. If I remove subclassing then it works.

    @denfromufa
    I do not understand. Well, I am not .NET pro. I compile with run.bat in Visual Studio 2015 Upd3 Developer Prompt and see only:
    C:\dev\job3>csc /target:library /debug:full class.cs
    Microsoft (R) Visual C# Compiler version 1.3.1.60616
    Copyright (C) Microsoft Corporation. All rights reserved.

    and class.dll is made somehow.

  6. testrunner123 commented on Jun 1, 2017

    @testrunner123
    testrunner123
    on Jun 1, 2017
    ContributorAuthor

    @vmuriart
    I tested and removing subclassing in sample code or just removing __namespace__ in adapter.py fixes the issue. Do you have an idea what this bug is ? Or may be it is fixed already??

  7. den-run-ai commented on Jun 1, 2017

    @den-run-ai
    den-run-ai
    on Jun 1, 2017
    Contributor

    I see that it is self-contained now - Aka namespace is in the same class.cs.

  8. testrunner123 commented on Jun 2, 2017

    @testrunner123
    testrunner123
    on Jun 2, 2017
    ContributorAuthor

    I nailed bug in src\runtime\typemananger.cs in CreateSubType(): method Runtime.XIncref() is not called, but PyObject is created and added to disposeList. Correct code:

    if (0 != Runtime.PyMapping_HasKey(py_dict, assemblyKey.Handle))
{
	var pyAssembly = new PyObject(Runtime.PyDict_GetItem(py_dict, assemblyKey.Handle));
	Runtime.XIncref(pyAssembly.Handle);
	...
}
...
if (0 != Runtime.PyMapping_HasKey(py_dict, namespaceKey.Handle))
{
	var pyNamespace = new PyObject(Runtime.PyDict_GetItem(py_dict, namespaceKey.Handle));
	Runtime.XIncref(pyNamespace.Handle);
...
}

    Now it doesn't crash anymore.

    Would be not easy to make regression test out of it. I realized that no crash if this code is completely in one .py. It must be spread among several modules to provoke garbage collector crash.

  9. vmuriart commented on Jun 2, 2017

    @vmuriart
    vmuriart
    on Jun 2, 2017
    Contributor

    Nice! The subclass tests might catch this. Even though it's intermittent I run it multiple times each version to check for it.

  10. testrunner123 commented on Jun 6, 2017

    @testrunner123
    testrunner123
    on Jun 6, 2017
    ContributorAuthor

    If it works in your tests then you can submit this patch to repo.

  11. den-run-ai commented on Jun 6, 2017

    @den-run-ai
    den-run-ai
    on Jun 6, 2017
    Contributor

    @testrunner123 would you like to submit a pull request? otherwise someone can do this for you and attribute the commit to you git commit --author="???"

  12. filmor
    added
    bug
    on Jun 6, 2017

  13. filmor commented on Jun 7, 2017

    @filmor
    filmor
    on Jun 7, 2017
    Member

    Hmm, we had quite a few of those. Maybe we should replace the current new PyObject by more explicit constructors like PyObject.FromPtrIncref, PyObject.BorrowFromPtr? I don't really have good names, though.

  14. den-run-ai commented on Jun 7, 2017

    @den-run-ai
    den-run-ai
    on Jun 7, 2017
    Contributor

    @filmor i agree about PyObject.FromPtrIncref, when would PyObject.BorrowFromPtr happen?

  15. den-run-ai
    added this to the 2.4.0 milestone on Jun 14, 2017

  16. filmor commented on Jul 11, 2017

    @filmor
    filmor
    on Jul 11, 2017
    Member

    I didn't think much about the names, I'll try to get a PR up with a proposal for explicit constructors.

  17. filmor
    modified the milestones: 2.4.0, 2.5.0 on Apr 12, 2019

  18. lostmsu commented on Apr 23, 2020

    @lostmsu
    lostmsu
    on Apr 23, 2020
    Member

    @filmor the fix seems to be merged. Can this bug be closed?

  19. filmor commented on Apr 23, 2020

    @filmor
    filmor
    on Apr 23, 2020
    Member

    I think so, yes.

  20. filmor
    closed this as completedon Apr 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions