Skip to content

Crash when there a "._xxx.pth" on macOS #113356

New issue
New issue
Closed
Closed
Crash when there a "._xxx.pth" on macOS#113356
Assignees
ronaldoussoren
Labels
OS-mactype-bugAn unexpected behavior, bug, or error

Description

@ronaldoussoren
ronaldoussoren
opened on Dec 21, 2023

Bug report

Bug description:

See https://discuss.python.org/t/venv-not-function-properly-in-macos-within-an-exfat-disk/26145 for the original report.

MacOS creates "._" files with additional metadata when there is metadata (such as extended attributes) that the filesystem doesn't support natively.

To reproduce (I did this on macOS 13 to match the forum message, likely also happens on 14):

  1. Create a disk image with the exFAT format and mount this

  2. Create a virtual environment on this volume and active this

  3. Install setup tools (which ships with a PTH file name distutils-precedence.pth)

  4. Use the "Get Info" context menu in the finder for the PTH file and add a note to the metadata

    There should now be a ._distutils-precedence.pth file

    Alternatively: use xattr -w name value distutils-precedence.pth to write an extended attribute.

  5. Start "python"
    This should fail with a UnicodeDecodeError in site.py

CPython versions tested on:

3.12

Operating systems tested on:

macOS

Linked PRs

Activity

  1. ronaldoussoren
    added
    type-bugAn unexpected behavior, bug, or error
    OS-mac
    on Dec 21, 2023
  2. ronaldoussoren
    self-assigned this
    on Dec 21, 2023
  3. ronaldoussoren
    added a commit that references this issue on Dec 21, 2023

    pythongh-113356: Ignore errors in "._ABC.pth"

    f6fe595

  4. quul commented on Dec 21, 2023

    @quul
    quul
    on Dec 21, 2023

    Thank you for your attention! I am the one who report this.
    I am now in macOS 14.2 (23C64), and I could reproduce the issue without manually tweaking(But obviously it doesn't effect the PR, just FYI).

    Python version: Python 3.11.6 (main, Nov 2 2023, 04:39:43) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin

    Reproduce steps:

    1. Format an extra disk with exFAT
    2. Create a venv with python3 -m venv venv
    3. Enter the venv via source venv/bin/activate
    4. Try to install anything with pip. For example pip install django, and then the error cames out

    The ._ file appears after step 2:

    ╰─$ ls -al venv/lib/python3.11/site-packages
total 4096
drwx------@ 1 lincw  staff  131072 Dec 21 22:46 .
drwx------@ 1 lincw  staff  131072 Dec 21 22:46 ..
-rwx------  1 lincw  staff    4096 Dec 21 22:46 .__distutils_hack
-rwx------  1 lincw  staff    4096 Dec 21 22:46 ._distutils-precedence.pth
-rwx------  1 lincw  staff    4096 Dec 21 22:46 ._pip
-rwx------  1 lincw  staff    4096 Dec 21 22:46 ._pip-23.3.1.dist-info
-rwx------  1 lincw  staff    4096 Dec 21 22:46 ._pkg_resources
-rwx------  1 lincw  staff    4096 Dec 21 22:46 ._setuptools
-rwx------  1 lincw  staff    4096 Dec 21 22:46 ._setuptools-68.2.2.dist-info
drwx------@ 1 lincw  staff  131072 Dec 21 22:46 _distutils_hack
-rwx------@ 1 lincw  staff     151 Dec 21 22:46 distutils-precedence.pth
drwx------@ 1 lincw  staff  131072 Dec 21 22:46 pip
drwx------@ 1 lincw  staff  131072 Dec 21 22:46 pip-23.3.1.dist-info
drwx------@ 1 lincw  staff  131072 Dec 21 22:46 pkg_resources
drwx------@ 1 lincw  staff  131072 Dec 21 22:46 setuptools
drwx------@ 1 lincw  staff  131072 Dec 21 22:46 setuptools-68.2.2.dist-info

    Sorry for anything bad since I'm not so good at English. :(

  5. ronaldoussoren commented on Dec 21, 2023

    @ronaldoussoren
    ronaldoussoren
    on Dec 21, 2023
    ContributorAuthor

    Thank you for your attention! I am the one who report this. I am now in macOS 14.2 (23C64), and I could reproduce the issue without manually tweaking(But obviously it doesn't effect the PR, just FYI).

    [...]

    Sorry for anything bad since I'm not so good at English. :(

    Thanks for the update. And don't worry about your English, it is perfectly clear.

  6. ronaldoussoren commented on Dec 22, 2023

    @ronaldoussoren
    ronaldoussoren
    on Dec 22, 2023
    ContributorAuthor

    @quul, What's the output of the following commands (before removing the "._" files):

    xattr -l venv/lib/python3.11/site-packages/distutils-precedence.pth
ls -le@ venv/lib/python3.11/site-packages/distutils-precedence.pth

    Not really important, but I'm curious what metadata is added on your system. I don't have an external disk lying around that I can format exFAT, otherwise I'd test myself.

  7. serhiy-storchaka commented on Dec 22, 2023

    @serhiy-storchaka
    serhiy-storchaka
    on Dec 22, 2023
    Member

    @ronaldoussoren, you perhaps can create a file system in a file and them mount it. At least it is easy to do on Linux:

    $ dd of=exfat.img if=/dev/null bs=1G count=0 seek=1
$ mkfs.exfat exfat.img
$ mkdir mountdir
$ sudo mount exfat.img mountdir

    We can ignore UnicodeDecodeError as well as OSError when open a pth file (although the code will be more complex to avoid leaks), but I think that it is reasonable to skip dot-files when list pth files at first place.

  8. ronaldoussoren commented on Dec 22, 2023

    @ronaldoussoren
    ronaldoussoren
    on Dec 22, 2023
    ContributorAuthor

    @ronaldoussoren, you perhaps can create a file system in a file and them mount it. At least it is easy to do on Linux:

    $ dd of=exfat.img if=/dev/null bs=1G count=0 seek=1
$ mkfs.exfat exfat.img
$ mkdir mountdir
$ sudo mount exfat.img mountdir

    We can ignore UnicodeDecodeError as well as OSError when open a pth file (although the code will be more complex to avoid leaks), but I think that it is reasonable to skip dot-files when list pth files at first place.

    I can reproduce the issue using a disk image, but only by manually settings some extended attributes. Users on the d.p.o thread I link to run into this without doing anything special. The only difference is that they use an external disk.

  9. quul commented on Dec 22, 2023

    @quul
    quul
    on Dec 22, 2023

    @quul, What's the output of the following commands (before removing the "._" files):

    xattr -l venv/lib/python3.11/site-packages/distutils-precedence.pth
ls -le@ venv/lib/python3.11/site-packages/distutils-precedence.pth

    Not really important, but I'm curious what metadata is added on your system. I don't have an external disk lying around that I can format exFAT, otherwise I'd test myself.

    I’m glad to do the test. However I just started my vocation and not carrying the mac with me.
    I’ll do this as soon as I come back to office(perhaps 1 week+)

  10. ronaldoussoren commented on Jan 18, 2024

    @ronaldoussoren
    ronaldoussoren
    on Jan 18, 2024
    ContributorAuthor

    This has been fixed by #113659, which will ignore all hidden PTH files, including those with a filename starting with a dot like the ._ files seen in this issue.

  11. ronaldoussoren
    closed this as completedon Jan 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

OS-mactype-bugAn unexpected behavior, bug, or error

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions