Skip to content

Difference in ressurrection behavior with __del__ in py2 vs. py3 #76558

Closed
Closed
Difference in ressurrection behavior with __del__ in py2 vs. py3#76558
Labels
3.7 (EOL)end of lifedocsDocumentation in the Doc dirtype-bugAn unexpected behavior, bug, or error

Description

@EricCousineau-TRI
EricCousineau-TRI
mannequin
opened on Dec 19, 2017
BPO 32377
Nosy @pitrou, @EricCousineau-TRI
PRs
  • bpo-32377: improve __del__ docs and fix mention about resurrection #4927
  • [3.6] bpo-32377: improve __del__ docs and fix mention about resurrection (GH-4927) #4929
    		•
    Files
  • revive_test.py: Repro example
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2017-12-19.19:00:56.948>
created_at = <Date 2017-12-19.16:29:59.034>
labels = ['type-bug', '3.7', 'docs']
title = 'Difference in ressurrection behavior with __del__ in py2 vs. py3'
updated_at = <Date 2020-04-09.15:25:33.816>
user = 'https://github.com/EricCousineau-TRI'

    bugs.python.org fields:

    activity = <Date 2020-04-09.15:25:33.816>
actor = 'Eric Cousineau'
assignee = 'docs@python'
closed = True
closed_date = <Date 2017-12-19.19:00:56.948>
closer = 'pitrou'
components = ['Documentation']
creation = <Date 2017-12-19.16:29:59.034>
creator = 'Eric Cousineau'
dependencies = []
files = ['47339']
hgrepos = []
issue_num = 32377
keywords = ['patch']
message_count = 9.0
messages = ['308660', '308661', '308662', '308664', '308665', '308667', '308671', '366057', '366068']
nosy_count = 3.0
nosy_names = ['pitrou', 'docs@python', 'Eric Cousineau']
pr_nums = ['4927', '4929']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32377'
versions = ['Python 3.6', 'Python 3.7']

    Activity

    1. EricCousineau-TRI commented on Dec 19, 2017

      @EricCousineau-TRI
      EricCousineau-TRImannequin
      on Dec 19, 2017
      MannequinAuthor

      Due to how PyObject_CallFinalizer is written in python3, __del__ will only ever be called once.

      In my use case, I am experimenting with a feature in pybind11 to prevent slicing with Python class instances that inherit from pybind11-C++ base classes, which involves detecting when an instance loses all reference in Python (Py_REFCNT(...) == 0) but still has reference in C++ (shared_ptr::count() > 0), and reviving the Python portion when this situation happens.

      In python2, I could do this without a hitch, as a resurrected object could have its __del__ method called multiple times (through tp_dealloc I believe?). But in python3, the object is marked with _PyGC_SET_FINALIZED(...), thus preventing __del__ from being called again.

      It'd be nice to either (a) somehow allow __del__ to be called naturally without too much fuss or, at the least, (b) have this reflected in the documentation:
      https://docs.python.org/3/reference/datamodel.html#object.__del__

      See attached revive_test. Example execution:

      $ python2 ./revive_test.py 
Revive
Destroy
[ Done ]

$ python3 ./revive_test.py 
Revive
[ Done ]
    2. EricCousineau-TRI
      assigned
      docspython
      on Dec 19, 2017
    3. EricCousineau-TRI
      added
      docsDocumentation in the Doc dir
      type-bugAn unexpected behavior, bug, or error
      on Dec 19, 2017

    4. pitrou commented on Dec 19, 2017

      @pitrou
      pitrou
      on Dec 19, 2017
      Member

      Thanks for the report. Apparently I forgot to update that piece of documentation when PEP-442 was implemented.

    5. pitrou
      added
      3.7 (EOL)end of life
      on Dec 19, 2017

    6. EricCousineau-TRI commented on Dec 19, 2017

      @EricCousineau-TRI
      EricCousineau-TRImannequin
      on Dec 19, 2017
      MannequinAuthor

      You're welcome, and thank you for the prompt response!

      I will say that it feels a tad odd to only have tp_finalize be called once for the entire lifetime of the object, while still having the option of it being resurrected.
      Is there any way to somehow "un-mark" the object to enable this workflow that I would like to have?

      My current hack is to call _PyGC_SET_FINALIZED(self, 0) - may I ask if there is a simpler way to do this?

    7. pitrou commented on Dec 19, 2017

      @pitrou
      pitrou
      on Dec 19, 2017
      Member

      Le 19/12/2017 à 17:46, Eric Cousineau a écrit :

      My current hack is to call _PyGC_SET_FINALIZED(self, 0) - may I ask if there is a simpler way to do this?

      Well... perhaps you could create another PyObject (it's just a wrapper,
      right?) since the old one doesn't have any outside references to it
      remaining.

      Note that calling __del__ only once is also how PyPy works:
      http://doc.pypy.org/en/latest/cpython_differences.html#differences-related-to-garbage-collection-strategies

      If there is some demand we could expose a higher-level spelling of
      _PyGC_SET_FINALIZED(self, 0).

    8. pitrou commented on Dec 19, 2017

      @pitrou
      pitrou
      on Dec 19, 2017
      Member

      I've proposed a documentation improvement in #4927 . Please chime in if you see have issues with it.

    9. pitrou commented on Dec 19, 2017

      @pitrou
      pitrou
      on Dec 19, 2017
      Member

      New changeset 4b96593 by Antoine Pitrou in branch 'master':
      bpo-32377: improve __del__ docs and fix mention about resurrection (bpo-4927)
      4b96593