Skip to content

TarFile.extract() suffers from hard links inside tarball #73798

Open
Open
TarFile.extract() suffers from hard links inside tarball#73798
Labels
3.7 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@JussiJudin
JussiJudin
mannequin
opened on Feb 21, 2017
BPO 29612
Nosy @gustaebel, @ethanfurman, @j0ack
PRs
  • bpo-29612 : Fix hard link creation in tar extraction when it points to themselves #5753
  • bpo-29612 : Fix hard link creation in tar extraction when it points to themselves #8700
  • [2.7] bpo-29612 : Fix hard link creation in tar extraction when it points to themselves #16958
    		•
    Files
  • tartest.py
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2017-02-21.10:10:46.770>
labels = ['3.7', '3.8', 'type-bug', 'library', '3.9']
title = 'TarFile.extract() suffers from hard links inside tarball'
updated_at = <Date 2020-01-19.18:55:54.762>
user = 'https://bugs.python.org/JussiJudin'

    bugs.python.org fields:

    activity = <Date 2020-01-19.18:55:54.762>
actor = 'zach.ware'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-02-21.10:10:46.770>
creator = 'Jussi Judin'
dependencies = []
files = ['46658']
hgrepos = []
issue_num = 29612
keywords = ['patch']
message_count = 7.0
messages = ['288284', '289360', '312235', '312345', '312900', '314588', '320140']
nosy_count = 6.0
nosy_names = ['guettli', 'lars.gustaebel', 'ethan.furman', 'Jussi Judin', 'jtrouverie', 'Larry Cook']
pr_nums = ['5753', '8700', '16958']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue29612'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

    Activity

    1. JussiJudin commented on Feb 21, 2017

      @JussiJudin
      JussiJudinmannequin
      on Feb 21, 2017
      MannequinAuthor

      I managed to create a tarball that brought out quite nasty behavior with tarfile.TarFile.extract() and tarfile.TarFile.extractall() functions when there are hard links inside a tarball that point to themselves with a file that is included in the tarball. In Python 2.7 it leads to an exception and with Python 3.4-3.6 it extracts the same file from the tarball multiple times.

      First we create a tarball that causes this behavior:

      $ mkdir -p tardata/1/2/3/4/5/6/7/8/9
$ dd if=/dev/zero of=tardata/1/2/3/4/5/6/7/8/9/zeros.data bs=1000000 count=500
# tar by default adds all directories recursively multiple times to the archive, but duplicates are created as hard links:
$ find tardata | xargs tar cvfz tardata.tar.gz

      Then let's extract the tarball with tarfile module
      Let following commands demonstrate what happens with the attached tartest.py file

      $ python2.7.13 tartest.py noskip tardata.tar.gz /tmp/tardata-python-2.7.13
...
tardata/1/2/3/4/5/6/7/8/9/zeros.data
...
tardata/1/2/3/4/5/6/7/8/9/zeros.data
Traceback (most recent call last):
  File "tartest.py", line 17, in <module>
    unarchive(skip, archive, dest)
  File "tartest.py", line 12, in unarchive
    tar_fd.extract(info, dest)
  File "python/2.7.13/lib/python2.7/tarfile.py", line 2118, in extract
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name))
  File "python/2.7.13/lib/python2.7/tarfile.py", line 2202, in _extract_member
    self.makelink(tarinfo, targetpath)
  File "python/2.7.13/lib/python2.7/tarfile.py", line 2286, in makelink
    os.link(tarinfo._link_target, targetpath)
OSError: [Errno 2] No such file or directory

      And with Python 3.6.0 (and earlier Python 3 series based Pythons that I have tested):

      $ time python3.6.0 tartest.py noskip tardata.tar.gz /tmp/tardata-python-3.6.0
...
tardata/1/2/3/4/5/6/7/8/9/zeros.data <-- this is extracted 11 times
...
real    0m42.747s
user    0m17.564s
sys     0m6.144s

      If we then make the tarfile skip extraction of hard links that point to themselves:

      $ time python3.6.0 tartest.py skip tardata.tar.gz /tmp/tardata-python-3.6.0
...
tardata/1/2/3/4/5/6/7/8/9/zeros.data <-- this is extracted once
...
Skipping tardata/1/2/3/4/5/6/7/8/9/zeros.data <-- skipped hard links 10 times
...
real    0m2.688s
user    0m1.816s
sys     0m0.532s

      From the used user CPU time it's obvious that there is happening a lot of unneeded decompression when we compare Python 3.6 results. If I use TarFile.extractall(), it behaves similarly as using TarFile.extract() individually on TarInfo objects. GNU tar seems to behave in such fashion that it skips over the extraction of the actual file data when it encounters this situation.

    2. JussiJudin
      added
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on Feb 21, 2017

    3. guettli commented on Mar 10, 2017

      @guettli
      guettlimannequin
      on Mar 10, 2017
      Mannequin

      I have the same issue on Python 2.7.12 (Ubuntu 16.04)

      I tried to execute tartest.py. But I could not find a way how to create the tar which is needed for tartest.py.

    4. LarryCook commented on Feb 16, 2018

      @LarryCook
      LarryCookmannequin
      on Feb 16, 2018
      Mannequin

      I recently hit this with Python 2.7.5 and 2.7.13. It has a very simple repro. Just specify the same file twice on the command line to tar (GNU 1.26):

      % tar cvf test.tar test.txt test.txt
      test.txt
      test.txt

      % tar tvf test.tar
      -rw-r--r-- root/root 24 2018-02-16 09:35 test.txt
      hrw-r--r-- root/root 0 2018-02-16 09:35 test.txt link to test.txt

      % python2.7
Python 2.7.5 (default, Aug  4 2017, 00:39:18) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import tarfile
>>> tarball = tarfile.open("test.tar")
>>> tarball.extractall()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/tarfile.py", line 2047, in extractall
    self.extract(tarinfo, path)
  File "/usr/lib64/python2.7/tarfile.py", line 2084, in extract
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name))
  File "/usr/lib64/python2.7/tarfile.py", line 2168, in _extract_member
    self.makelink(tarinfo, targetpath)
  File "/usr/lib64/python2.7/tarfile.py", line 2252, in makelink
    os.link(tarinfo._link_target, targetpath)
OSError: [Errno 2] No such file or directory
>>>

    5. j0ack commented on Feb 19, 2018

      @j0ack
      j0ackmannequin
      on Feb 19, 2018
      Mannequin

      Is there anybody working on this issue or can I create a branch concerning it ?

    6. j0ack commented on Feb 26, 2018

      @j0ack
      j0ackmannequin
      on Feb 26, 2018
      Mannequin

      I created a PR for this issue for Python 2.7 (https://github.com/python/cpython/pull/5753/files).

      I just skip the link creation if the target path is equals to the link target. I don't see any corner case where this would be an unwanted behavior.

      I am not sure either I should also create an unit test for this behavior.

    7. j0ack commented on Mar 28, 2018

      @j0ack
      j0ackmannequin
      on Mar 28, 2018
      Mannequin

      Anyone for a review ?

    8. j0ack commented on Jun 21, 2018

      @j0ack
      j0ackmannequin
      on Jun 21, 2018
      Mannequin

      Travis build failed for a reason unrelated to my changes. I relaunched it using an empty commit.

      If anyone could validate my changes I would rebase to validate my work.

    9. zware
      added
      3.7 (EOL)end of life
      3.8 (EOL)end of life
      3.9 (EOL)end of life
      on Jan 19, 2020
    10. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    11. serhiy-storchaka
      added this to Tarfile issues 🗄️on May 20, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

      Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions