Skip to content

os.getgrouplist() can fail on macOS if the user has more than 17 groups #84195

New issue
New issue
Closed
Closed
os.getgrouplist() can fail on macOS if the user has more than 17 groups#84195
Labels
3.7 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifeOS-mactestsTests in the Lib/test dirtype-featureA feature request or enhancement

Description

@corona10
corona10
opened on Mar 19, 2020
BPO 40014
Nosy @ronaldoussoren, @vstinner, @ned-deily, @zooba, @corona10, @miss-islington, @shihai1991
PRs
  • bpo-40014: Skip os.getgrouplist related test if OSError is raised. #19075
  • bpo-40014: test.pythoninfo catchs os.getgrouplist() error #19117
  • bpo-40014: Fix os.getgrouplist() failure on macOS with many groups #19118
  • [3.8] bpo-40014: Fix os.getgrouplist() on macOS (GH-19118) #19123
  • [3.7] bpo-40014: Fix os.getgrouplist() on macOS (GH-19118) #19124
  • bpo-40014: Fix os.getgrouplist() #19126
  • [3.8] bpo-40014: Fix os.getgrouplist() (GH-19126) #19143
  • [3.7] bpo-40014: Fix os.getgrouplist() (GH-19126) #19144
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2020-03-24.17:51:27.264>
created_at = <Date 2020-03-19.16:04:54.967>
labels = ['OS-mac', '3.7', '3.8', '3.9', 'type-feature', 'tests']
title = 'os.getgrouplist() can fail on macOS if the user has more than 17 groups'
updated_at = <Date 2020-03-24.17:51:27.264>
user = 'https://github.com/corona10'

    bugs.python.org fields:

    activity = <Date 2020-03-24.17:51:27.264>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-03-24.17:51:27.264>
closer = 'vstinner'
components = ['macOS', 'Tests']
creation = <Date 2020-03-19.16:04:54.967>
creator = 'corona10'
dependencies = []
files = []
hgrepos = []
issue_num = 40014
keywords = ['patch']
message_count = 17.0
messages = ['364607', '364608', '364610', '364615', '364862', '364864', '364870', '364880', '364884', '364885', '364888', '364889', '364904', '364950', '364955', '364956', '364957']
nosy_count = 7.0
nosy_names = ['ronaldoussoren', 'vstinner', 'ned.deily', 'steve.dower', 'corona10', 'miss-islington', 'shihai1991']
pr_nums = ['19075', '19117', '19118', '19123', '19124', '19126', '19143', '19144']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue40014'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

    Activity

    1. corona10 commented on Mar 19, 2020

      @corona10
      corona10
      on Mar 19, 2020
      MemberAuthor

      example: https://github.com/python/cpython/pull/19073/checks?check_run_id=519539592

      I suggest to not to add information for os.getgrouplist
      if the OSError is raised.

    2. corona10
      added
      testsTests in the Lib/test dir
      type-featureA feature request or enhancement
      on Mar 19, 2020

    3. corona10 commented on Mar 19, 2020

      @corona10
      corona10
      on Mar 19, 2020
      MemberAuthor

      or adding <OSError> might be great just like getpwuid

    4. vstinner commented on Mar 19, 2020

      @vstinner
      vstinner
      on Mar 19, 2020
      Member

      These errors are really strange.

      Steve Dower: are you aware of anything recent change on the macOS job of Azure Pipelines / GH Actions.

      pythoninfo:
      ---
      os.uname: posix.uname_result(sysname='Darwin', nodename='Mac-1422.local', release='19.3.0', version='Darwin Kernel Version 19.3.0: Thu Jan 9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64', machine='x86_64')

      platform.architecture: 64bit

      platform.platform: macOS-10.15.3-x86_64-i386-64bit
      ---

      example: https://github.com/python/cpython/pull/19073/checks?check_run_id=519539592

      ./python.exe -m test.pythoninfo
ERROR: collect_pwd() failed
Traceback (most recent call last):
  File "/Users/runner/runners/2.165.2/work/cpython/cpython/Lib/test/pythoninfo.py", line 761, in collect_info
    collect_func(info_add)
  File "/Users/runner/runners/2.165.2/work/cpython/cpython/Lib/test/pythoninfo.py", line 336, in collect_pwd
    groups = os.getgrouplist(entry.pw_name, entry.pw_gid)
OSError: [Errno 0] Error

      Collection failed: exit with error

      PR 19075

      test_posix.test_getgrouplist() failed with:

      ======================================================================
      ERROR: test_getgrouplist (test.test_posix.PosixTester)
      ----------------------------------------------------------------------

      Traceback (most recent call last):
  File "/Users/runner/runners/2.165.2/work/cpython/cpython/Lib/test/test_posix.py", line 1028, in test_getgrouplist
    self.assertIn(group, posix.getgrouplist(user, group))
OSError: [Errno 25] Inappropriate ioctl for device
    5. vstinner
      added
      3.9 (EOL)end of life
      OS-mac
      on Mar 19, 2020
    6. vstinner
      changed the title [-]os.getgrouplist can raise OSError during the Display build info[/-] [+]os.getgrouplist() fails on macOS of GH Actions (Azure)[/+] on Mar 19, 2020
    7. vstinner
      added
      3.9 (EOL)end of life
      OS-mac
      on Mar 19, 2020
    8. vstinner
      changed the title [-]os.getgrouplist can raise OSError during the Display build info[/-] [+]os.getgrouplist() fails on macOS of GH Actions (Azure)[/+] on Mar 19, 2020

    9. ned-deily commented on Mar 19, 2020

      @ned-deily
      ned-deily
      on Mar 19, 2020
      Member

      This error has to do with the number of groups a particular user is associated with. We’ve squashed similar bugs in the past but it looks like something has changed again in recent releases of macOS. It’s not Azure specific.

    10. ned-deily
      changed the title [-]os.getgrouplist() fails on macOS of GH Actions (Azure)[/-] [+]os.getgrouplist() can fail on macOS[/+] on Mar 19, 2020

    11. 10 remaining items

    12. ned-deily
      changed the title [-]os.getgrouplist() fails on macOS 10.15 (Catalina) if the user has more than 17 groups[/-] [+]os.getgrouplist() can fail on macOS if the user has more than 17 groups[/+] on Mar 23, 2020
    13. ned-deily
      changed the title [-]os.getgrouplist() fails on macOS 10.15 (Catalina) if the user has more than 17 groups[/-] [+]os.getgrouplist() can fail on macOS if the user has more than 17 groups[/+] on Mar 23, 2020

    14. vstinner commented on Mar 23, 2020

      @vstinner
      vstinner
      on Mar 23, 2020
      Member

      Most likely this hasn't been an issue before is that 10.15 creates more groups for system services than the earlier versions did.

      Aha, interesting. I updated my PR to remove mentions of the macOS version number. I just wrote "on macOS". Yeah, I wouldn't be surprised that the function always failed.

      --

      I'm not sure that the Linux implementation is correct. It rely on errno, whereas Linux manual page of getgrouplist() doesn't say anything about errno being set on error.

      Moreover, on Linux, getgrouplist() can also fail with -1 if the group list is too small. In that case, ngroups is updated to the number of groups and so can be used to enlarge the list!

    15. vstinner commented on Mar 23, 2020

      @vstinner
      vstinner
      on Mar 23, 2020
      Member

      test.pythoninfo of the macOS job on the PR:

      os.getgrouplist: 20, 0, 12, 61, 79, 80, 81, 98, 264, 399, 701, 33, 100, 204, 250, 395, 398, 400
      os.getgroups: 20, 0, 12, 61, 79, 80, 81, 98, 264, 399, 701, 33, 100, 204, 250, 395, 398, 400

      So yeah, there are 18 groups ;-)

    16. vstinner commented on Mar 23, 2020

      @vstinner
      vstinner
      on Mar 23, 2020
      Member

      New changeset 8ec7370 by Victor Stinner in branch 'master':
      bpo-40014: Fix os.getgrouplist() on macOS (GH-19118)
      8ec7370

    17. miss-islington commented on Mar 23, 2020

      @miss-islington
      miss-islington
      on Mar 23, 2020
      Contributor

      New changeset 21bee0b by Miss Islington (bot) in branch '3.8':
      bpo-40014: Fix os.getgrouplist() on macOS (GH-19118)
      21bee0b

    18. miss-islington commented on Mar 23, 2020

      @miss-islington
      miss-islington
      on Mar 23, 2020
      Contributor

      New changeset 1cdc61c by Miss Islington (bot) in branch '3.7':
      bpo-40014: Fix os.getgrouplist() on macOS (GH-19118)
      1cdc61c

    19. vstinner commented on Mar 23, 2020

      @vstinner
      vstinner
      on Mar 23, 2020
      Member

      Moreover, on Linux, getgrouplist() can also fail with -1 if the group list is too small. In that case, ngroups is updated to the number of groups and so can be used to enlarge the list!

      I tested by manually initializing ngroups to a value way lower than MAX_GROUPS (65536 on my Fedora 31): os.getgrouplist() raises OSError() with a random errno, getgrouplist() doesn't set errno on failure.

      I wrote PR 19126 to fix the issue on all platforms. The glibc implementation is detected and used: use updated ngroups value (only if it's larger).

    20. vstinner commented on Mar 24, 2020

      @vstinner
      vstinner
      on Mar 24, 2020
      Member

      New changeset f5c7cab by Victor Stinner in branch 'master':
      bpo-40014: Fix os.getgrouplist() (GH-19126)
      f5c7cab

    21. miss-islington commented on Mar 24, 2020

      @miss-islington
      miss-islington
      on Mar 24, 2020
      Contributor

      New changeset 5753fc6 by Miss Islington (bot) in branch '3.7':
      bpo-40014: Fix os.getgrouplist() (GH-19126)
      5753fc6

    22. miss-islington commented on Mar 24, 2020

      @miss-islington
      miss-islington
      on Mar 24, 2020
      Contributor

      New changeset af6fd1f by Miss Islington (bot) in branch '3.8':
      bpo-40014: Fix os.getgrouplist() (GH-19126)
      af6fd1f

    23. vstinner commented on Mar 24, 2020

      @vstinner
      vstinner
      on Mar 24, 2020
      Member

      Ned: so Linux was also impacted, but Linux only has an issue with more than 65536 groups :-D macOS MAX_GROUPS is now only 16 (Python uses MAX_GROUPS+1)! Maybe MAX_GROUPS was reduced recently. I'm not sure.

      Anyway, os.getgrouplist() does no longer depend on MAX_GROUPS hardcoded limit, but grow the group list dynamically. It should now work with any number of groups an all platforms (which provide the getgrouplist() function ;-)) on Python 3.7, 3.8 and master branches.

      Thanks Dong-hee Na for the bug report and your PR.

    24. vstinner
      closed this as completedon Mar 24, 2020
    25. vstinner
      closed this as completedon Mar 24, 2020
    26. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifeOS-mactestsTests in the Lib/test dirtype-featureA feature request or enhancement

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions