Clean issue when generator not exhausted (garbage collector related?) #69255

Activity

1. guilimo commented on Sep 11, 2015

   

   guilimomannequin

   on Sep 11, 2015

   MannequinAuthor

   More actions

   Hello!

   I experienced a strange behavior when using a generator, with some code encapsulated in a try/finally clause or a context manager.

   If the generator is not exhausted when we leave the script (because normal end of script, uncatched exception, etc...), I expect an internal mechanism to execute properly the finally clause (or __exit__ if context manager).
   However, in some specific cases, it seems that this mechanism does not work as expected.

   Example
   =======

   Consider the following code:

   import time
   
   def create_generator():
       try:
           yield
       finally:
           time.sleep(2)
   
   ct = create_generator()
   ct.next()

   If you try to execute it, you will get a:
   "Exception AttributeError: "'NoneType' object has no attribute 'sleep'" in <generator object create_generator at 0x7f04ad62c0f0> ignored"

   Description
   ===========

   My understanding is the following (but I may be wrong):
   At the end of the script, the garbage collector automatically calls the close() method of the generator. As a result, GeneratorExit is raised from the "yield", the finally clause is executed, but for some reason, time does not exist anymore (already collected by the GC?).
   If you try just a print "xxx" instead of the time.sleep, it seems that there is not any problem.

   Important note:
   ===============

   An other very strange thing:
   It seems that if I use an other name than "ct" for the generator, the same exact code runs flawlessly...

   You can find attached 3 examples (with try/finally, with a context manager, and with an other name for the generator).

   I also found this ticket where some discussion may be related to my situation, even if not describing exactly my current problem:
   https://bugs.python.org/issue25014
2. guilimo

   added

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on Sep 11, 2015
3. vadmium commented on Sep 11, 2015

   

   vadmium

   on Sep 11, 2015

   Member

   More actions

   You are basically right that it is garbage collection which causes your generator instance to be cleaned up, and in this instance it is happening after “time.sleep” has been set to None. It could also happen in the order, which would explain why changing names changes the behaviour you see.

   Why the interpreter has to set module globals to None I’ve never quite understood. Maybe it is to guarantee that modules involved in garbage cycles with __del__() can still be partly cleaned up, otherwise such a cycle could easily keep lots of modules alive via globals.

   This is not documented very obviously either. The best I can find at the moment is the warning under <https://docs.python.org/2/reference/datamodel.html#object.\_\_del__\>, which says “when __del__() is invoked in response to a module being deleted, [globals may be] in the process of being torn down”. The same applies to generator cleanup code. Looking through Python code you will often find workarounds such as prefixing globals with an underscore or caching references in non-global locations.

   My recommended workaround though is to avoid relying on garbage collection as much as possible, and instead use a try / finally or “with” block. Then you know exactly when your generator will be cleaned up:

   ct = create_generator()
   try:
       ct.next()
   finally:
       ct.close()
   
   from contextlib import closing
   with closing(create_generator()) as ct:
       ct.next()
4. vadmium

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   and removed

   type-crashA hard crash of the interpreter, possibly with a core dumpA hard crash of the interpreter, possibly with a core dump

   on Sep 11, 2015
5. bitdancer commented on Sep 11, 2015

   

   bitdancer

   on Sep 11, 2015

   Member

   More actions

   Yes, it is (was) to break otherwise unbreakable GC cycles involving __del__. Many improvements have been made to these algorithms in python3, such that __del__ methods no longer create unbreakable cycles, although I believe a few cases still remain. Since some python interpreters do not do GC the way CPython does, it is always best to not depend on GC cleanup but instead to be explicit.
6. bitdancer

   closed this as completedon Sep 11, 2015
7. bitdancer

   added

   invalid

   on Sep 11, 2015
8. ezio-melotti

   transferred this issue fromon Apr 10, 2022