mock_open is not compatible with read(n) (and pickle.load) #76036

Activity

1. RonRothman commented on Oct 23, 2017

   

   RonRothmanmannequin

   on Oct 23, 2017

   MannequinAuthor

   More actions

   mock.mock_open works as expected when reading the entire file (read()) or when reading a single line (readline()), but it seems to not support reading a number of bytes (read(n)).

   These work as expected:

       from mock import mock_open, patch
   
       # works: consume entire "file"
       with patch('__main__.open', mock_open(read_data='bibble')) as m:
           with open('foo') as h:
               result = h.read()

   assert result == 'bibble'  # ok
   

       # works: consume one line
       with patch('__main__.open', mock_open(read_data='bibble\nbobble')) as m:
           with open('foo') as h:
               result = h.readline()

   assert result == 'bibble\n'  # ok
   

   But trying to read only a few bytes fails--mock_open returns the entire read_data instead:

       # consume first 3 bytes of the "file"
       with patch('__main__.open', mock_open(read_data='bibble')) as m:
           with open('foo') as h:
               result = h.read(3)

   assert result == 'bib', 'result of read: {}'.format(result)  # fails
   

   Output:

       Traceback (most recent call last):
         File "/tmp/t.py", line 25, in <module>
           assert result == 'bib', 'result of read: {}'.format(result)
       AssertionError: result of read: bibble

   The unfortunate effect of this is that mock_open cannot be used with pickle.load.

       with open('/path/to/file.pkl', 'rb') as f:
           x = pickle.load(f)  # this requires f.read(1) to work
2. RonRothman

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Oct 23, 2017
3. RonRothman commented on Oct 23, 2017

   

   RonRothmanmannequin

   on Oct 23, 2017

   MannequinAuthor

   More actions

   Confirmed that the behavior exists in Python 3.6 as well.
4. rhettinger

   assigned

   voidspace

   on Oct 24, 2017
5. tirkarthi commented on Dec 16, 2018

   

   tirkarthi

   on Dec 16, 2018

   Member

   More actions

   Internally mock_open implementation uses line based iteration [0] to keep track of the state change between read calls. So readline too ignores the argument. There is bpo-25690 for an alternate mock_open implementation but the code change is large and adds a lot of features. A simpler approach would be to use StringIO or BytesIO to keep track of state changes and they provide read, readline and readlines API. mock_open docs mentions about using a customized mock for complex cases but I don't know if worthy enough to make this enhancement given the internal implementation change to support the API or to document this behavior.

   Looking further there also seems to be a test case for it [1] which will fail if this is fixed since this returns all characters instead of first 10 like using open().read(10).

       def test_mock_open_read_with_argument(self):
           # At one point calling read with an argument was broken
           # for mocks returned by mock_open
           some_data = 'foo\nbar\nbaz'
           mock = mock_open(read_data=some_data)
           self.assertEqual(mock().read(10), some_data)

   $ echo -n 'foo\nbar\nbaz' > /tmp/a.txt
   $ ./python.exe
   Python 3.8.0a0 (heads/master:f5107dfd42, Dec 16 2018, 13:41:57)
   [Clang 7.0.2 (clang-700.1.81)] on darwin
   Type "help", "copyright", "credits" or "license" for more information.
   >>> with open('/tmp/a.txt') as f:
   ...     actual = f.read(10)
   ...     actual, len(actual)
   ...
   ('foo\nbar\nba', 10)
   >>> with open('/tmp/a.txt') as f:
   ...     from unittest.mock import mock_open
   ...     mock = mock_open(read_data=f.read())
   ...     mock_data = mock().read(10)
   ...     mock_data, len(mock_data)
   ...
   ('foo\nbar\nbaz', 11)

   [0]

   cpython/Lib/unittest/mock.py

   Line 2349 in f5107df

   def _iterate_read_data(read_data):

   
   [1]

   cpython/Lib/unittest/test/testmock/testwith.py

   Line 284 in f5107df

   def test_mock_open_read_with_argument(self):

   See More
6. tirkarthi

   added

   3.7 (EOL)end of lifeend of life

   3.8 (EOL)end of lifeend of life

   on Dec 16, 2018
7. remilapeyre commented on Jan 11, 2019

   

   remilapeyremannequin

   on Jan 11, 2019

   Mannequin

   More actions

   The off-by-one error in a test added for an unrelated issue (bpo-17467) makes me think @michael.foord just made a mistake in the test.

   mock_open docs mentions about using a customized mock for complex cases
   I think it's more for complex things like fetching data using a callback for example, this seems like a normal use case for mock_open(). Even more so, leaving the behavior as it is now may lead someone to think there is a bug in its code.

   I opened a new PR to fix this issue: #11521
8. cjw296 commented on May 7, 2019

   

   cjw296

   on May 7, 2019

   Contributor

   More actions

   New changeset 11a8832 by Chris Withers (Rémi Lapeyre) in branch 'master':
   bpo-31855: unittest.mock.mock_open() results now respects the argument of read([size]) (GH-11521)
   11a8832
9. cjw296 commented on May 7, 2019

   

   cjw296

   on May 7, 2019

   Contributor

   More actions

   New changeset a6516f8 by Chris Withers (Miss Islington (bot)) in branch '3.7':
   bpo-31855: unittest.mock.mock_open() results now respects the argument of read([size]) (GH-11521) (bpo-13152)
   a6516f8
10. tirkarthi commented on May 7, 2019

    

    tirkarthi

    on May 7, 2019

    Member

    More actions

    PR was merged and backported to 3.7. So I am closing this as fixed. Thanks Rémi for the patch.
11. tirkarthi

    closed this as completedon May 7, 2019
12. ezio-melotti

    transferred this issue fromon Apr 10, 2022