PyMem_RawMalloc(): update also sys.getallocatedblocks() in debug mode #71037

Activity

1. vstinner commented on Apr 26, 2016

   

   vstinner

   on Apr 26, 2016

   MemberAuthor

   More actions

   I modified PyMem_Malloc() to use the pymalloc allocator in the issue bpo-26249. This change helped to find a memory leak in test_format that I introduced in Python 3.6:
   http://bugs.python.org/issue26249#msg264174

   This memory leak gave me an idea: PyMem_RawMalloc() should also update sys.getallocatedblocks() (number of currently allocated blocks). It would help to find memory leaks using "python -m test -R 3:3" in extension modules using PyMem_RawMalloc() (and not PyMem_Malloc() or PyObject_Malloc()).

   Attached patch uses an atomic variable _Py_AllocatedBlocks, but only in debug mode. I chose to only change the variable in debug mode to:

   • not impact performances
   • I don't know if atomic variables are well supported (especially the "var++" operation)
   • I don't know yet the impact of this change (how sys.getallocatedblocks() is used).

   (The patch would be simpler if the release mode would also be impacted.)

   The patch changes _PyObject_Alloc() and _PyObject_Free() in debug mode to only account allocations directly made by pymalloc, to let PyMem_RawMalloc() and PyMem_RawFree() update the _Py_AllocatedBlocks variable. In release mode, _PyObject_Alloc() and _PyObject_Free() are responsible to update the _Py_AllocatedBlocks variable for allocations delegated to PyMem_RawMalloc() and PyMem_RawFree().
2. vstinner

   added

   type-featureA feature request or enhancementA feature request or enhancement

   on Apr 26, 2016
3. vstinner commented on Apr 26, 2016

   

   vstinner

   on Apr 26, 2016

   MemberAuthor

   More actions

   The expected side effect of this change is that hunting memory leaks in regrtest (python -m test) may get more noise.

   I ran the Python test suite using "./python -m test -R 3:3": test_nntplib, test_tools and test_unittest failed.

   Run alone, test_tools still fails:
   -------------

   $ ./python -m test -R 3:3 test_tools
   0:00:00 [1/1] test_tools
   beginning 6 repetitions
   123456
   ......
   test_tools leaked [0, 5, 20] references, sum=25
   test_tools leaked [0, 1, 4] memory blocks, sum=5
   test_tools took 44 sec
   1 test failed:
       test_tools
   Total duration: 0:00:45

   ---

   But if I run test_tools one more time, it doesn't fail anymore...

   Example of two sequential runs using two processes:
   -------------

   $ ./python -m test -R 3:3 test_tools
   Run tests sequentially
   0:00:00 [1/1] test_tools
   beginning 6 repetitions
   123456
   ......
   test_tools took 44 sec
   1 test OK.
   Total duration: 0:00:44
   
   $ ./python -m test -R 3:3 test_tools
   Run tests sequentially
   0:00:00 [1/1] test_tools
   beginning 6 repetitions
   123456
   ......
   test_tools leaked [2, 0, 10] references, sum=12
   test_tools leaked [0, 0, 3] memory blocks, sum=3
   test_tools took 43 sec
   1 test failed:
       test_tools
   Total duration: 0:00:44

   ---

   test_nntplib doesn't fail anymore when run alone.

   Oh, and test_unittest failure doesn't seem related to my change: "./python -m test -R 3:3 test_unittest" already fails without my change.
4. vstinner commented on May 20, 2016

   

   vstinner

   on May 20, 2016

   MemberAuthor

   More actions

   I'm not convinced myself by this change, it may make detection of memory leaks much more tricky for a little gain. I prefer to reject my own issue ;-)
5. vstinner

   closed this as completedon May 20, 2016
6. ezio-melotti

   transferred this issue fromon Apr 10, 2022