Skip to content

Build does not work with closed stdin on NetBSD #77565

Closed

Description

@MartinHusemann
BPO 33384
Nosy @vstinner, @0-wiz-0, @xdegaye, @MartinHusemann, @LeoT
Superseder
  • bpo-30225: EBADF error on x86 Tiger 3.x buildbot
  • Files
  • patch-Python_pylifecycle.c: Use fstat(2) instead of dup(2) on NetBSD too. The behaviour are the same of the one niced in bpo-30225
  • is_valid_fd.c: is_valid_fd() from Python 3.6.5 (Python/pylifecycle.c)
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-10-22.23:30:02.974>
    created_at = <Date 2018-04-29.10:16:35.561>
    labels = ['type-feature', 'build']
    title = 'Build does not work with closed stdin on NetBSD'
    updated_at = <Date 2019-10-24.10:39:12.784>
    user = 'https://github.com/MartinHusemann'

    bugs.python.org fields:

    activity = <Date 2019-10-24.10:39:12.784>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-10-22.23:30:02.974>
    closer = 'vstinner'
    components = ['Build']
    creation = <Date 2018-04-29.10:16:35.561>
    creator = 'MartinHusemann'
    dependencies = []
    files = ['47604', '47605']
    hgrepos = []
    issue_num = 33384
    keywords = []
    message_count = 9.0
    messages = ['315886', '316183', '317110', '317114', '317117', '317121', '355179', '355321', '355323']
    nosy_count = 5.0
    nosy_names = ['vstinner', 'wiz', 'xdegaye', 'MartinHusemann', 'leot']
    pr_nums = []
    priority = 'normal'
    resolution = 'duplicate'
    stage = 'resolved'
    status = 'closed'
    superseder = '30225'
    type = 'enhancement'
    url = 'https://bugs.python.org/issue33384'
    versions = ['Python 3.6']

    Activity

    1. MartinHusemann commented on Apr 29, 2018

      MartinHusemannmannequin
      MannequinAuthor

      When building python extensions in the background w/o stdin (and stderr and stdout redirected to a log file), the invocation of setup.py fails.

      Normal build in a shell:

      Example from pyexpat:

      /usr/pkg/bin/python3.6 setup.py build
      running build
      running build_ext
      building 'pyexpat' extension
      creating build
      creating build/temp.netbsd-8.99.14-amd64-3.6
      creating build/temp.netbsd-8.99.14-amd64-3.6/Modules
      gcc -DNDEBUG -O2 -pipe -D_FORTIFY_SOURCE=2 -I/usr/include -I/usr/pkg/include -O2 -pipe -D_FORTIFY_SOURCE=2 -I/usr/include -I/usr/pkg/include/python3.6 -I/usr/include -I/usr/pkg/include/python3.6 -fPIC -DHAVE_EXPAT_H -I/work/pkgobj/textproc/py-expat/work/.buildlink/include -I/usr/pkg/include/python3.6 -c Modules/pyexpat.c -o build/temp.netbsd-8.99.14-amd64-3.6/Modules/pyexpat.o
      creating build/lib.netbsd-8.99.14-amd64-3.6
      gcc -pthread -shared -L. -L/usr/lib -Wl,-R/usr/lib -L/usr/pkg/lib -Wl,-R/usr/pkg/lib -L/usr/X11R7/lib -Wl,-R/usr/X11R7/lib -L/usr/lib -Wl,-R/usr/lib -L/usr/pkg/lib -Wl,-R/usr/pkg/lib -L/usr/X11R7/lib -Wl,-R/usr/X11R7/lib -L/usr/lib -Wl,-R/usr/lib -Wl,-R/usr/pkg/lib -O2 -pipe -D_FORTIFY_SOURCE=2 -I/usr/include -I/usr/pkg/include/python3.6 -I/usr/include -I/usr/pkg/include/python3.6 build/temp.netbsd-8.99.14-amd64-3.6/Modules/pyexpat.o -L/work/pkgobj/textproc/py-expat/work/.buildlink/lib -L/usr/pkg/lib -Wl,-R/work/pkgobj/textproc/py-expat/work/.buildlink/lib -lexpat -lpython3.6 -o build/lib.netbsd-8.99.14-amd64-3.6/pyexpat.so

      But if invoked as

      ( sleep 15; /usr/pkg/bin/python3.6 setup.py build ) >/tmp/log 2>&1 &
      ^D

      and then watching /tmp/log from another session:

      Fatal Python error: Py_Initialize: can't initialize sys standard streams
      OSError: [Errno 9] Bad file descriptor

      Current thread 0x000075b4e4cee800 (most recent call first):
      [1] Abort trap (core dumped) /usr/pkg/bin/pyt...

    2. xdegaye commented on May 4, 2018

      xdegayemannequin
      Mannequin

      I cannot reproduce the problem with Python 3.6.5, building pyephem and with the command:

      ( sleep 15; python setup.py build ) >foo.log 2>&1

    3. MartinHusemann commented on May 19, 2018

      MartinHusemannmannequin
      MannequinAuthor

      You need to exit the parent shell, to get the original stdin revoke(2)'d.
      That is: the Ctrl-D in the original descritpion is not line noise. Sorry, should have been more explicit (or used "exit" or something instead)

    4. leot commented on May 19, 2018

      leotmannequin
      Mannequin

      A simpler way to reproduce that (probably this problem is not limited to
      building but to sys module):

      (sleep 10; python3.6 -c 'import sys' >/tmp/log 2>&1) & exit
      

      And in `/tmp/log':

      Fatal Python error: Py_Initialize: can't initialize sys standard streams
      OSError: [Errno 9] Bad file descriptor
       
      Current thread 0x00007226e0ab3800 (most recent call first):
      
    5. leot commented on May 19, 2018

      leotmannequin
      Mannequin

      And here the backtrace of the corresponding core (this is on
      NetBSD/amd64 8.99.15 with lang/python36 package of today pkgsrc-current):

      % gdb -core python3.6.core `which python3.6`
      Reading symbols from /usr/pkg/bin/python3.6...done.
      [New process 1]
      Core was generated by `python3.6'.
      Program terminated with signal SIGABRT, Aborted.
      #0  0x000076255932924a in _lwp_kill () from /usr/lib/libc.so.12
      (gdb) bt
      #0  0x000076255932924a in _lwp_kill () from /usr/lib/libc.so.12
      #1  0x0000762559328e65 in abort () at /usr/src/lib/libc/stdlib/abort.c:74
      #2  0x000076255a749d33 in Py_FatalError (msg=msg@entry=0x76255a7dea20 "Py_Initialize: can't initialize sys standard streams")
          at [Python/pylifecycle.c:1462](https://github.com/python/cpython/blob/main/Python/pylifecycle.c#L1462)
      #3  0x000076255a74af4c in _Py_InitializeEx_Private (install_sigs=1, install_importlib=1) at [Python/pylifecycle.c:450](https://github.com/python/cpython/blob/main/Python/pylifecycle.c#L450)
      #4  0x000076255a764902 in Py_Main (argc=argc@entry=3, argv=argv@entry=0x76255ab01060) at [Modules/main.c:700](https://github.com/python/cpython/blob/main/Modules/main.c#L700)
      #5  0x0000000000400e2f in main (argc=3, argv=<optimized out>) at ./Programs/python.c:69
      (gdb) f 4
      #4  0x000076255a764902 in Py_Main (argc=argc@entry=3, argv=argv@entry=0x76255ab01060) at [Modules/main.c:700](https://github.com/python/cpython/blob/main/Modules/main.c#L700)
      700         Py_Initialize();
      (gdb) f 3
      #3  0x000076255a74af4c in _Py_InitializeEx_Private (install_sigs=1, install_importlib=1) at [Python/pylifecycle.c:450](https://github.com/python/cpython/blob/main/Python/pylifecycle.c#L450)
      450             Py_FatalError(
      (gdb) list
      445         if (_PyTraceMalloc_Init() < 0)
      446             Py_FatalError("Py_Initialize: can't initialize tracemalloc");
      447
      448         initmain(interp); /* Module __main__ */
      449         if (initstdio() < 0)
      450             Py_FatalError(
      451                 "Py_Initialize: can't initialize sys standard streams");
      452
      453         /* Initialize warnings. */
      454         if (PySys_HasWarnOptions()) {
      

      Please let me know if any further information is needed and I will try
      to help!

      Thank you!

    6. leot commented on May 19, 2018

      leotmannequin
      Mannequin

      After testing is_valid_fd() (from Python/pylifecycle.c) separately
      (an `is_valid_fd.c' file will be attached to ease reproduction) I think
      that also NetBSD is affected by bpo-30225.

      Using dup(2) (at is currently done in NetBSD):

      % cc -o ivf is_valid_fd.c
      % sleep 5 && ./ivf > /tmp/log & exit
      [... the terminal is closed via ^D ...]
      % cat /tmp/log
      0: 1
      1: 1
      2: 1
      

      Using fstat(2):

      % cc -DUSE_FSTAT -o ivf is_valid_fd.c
      % sleep 5 && ./ivf > /tmp/log & exit
      [... the terminal is closed via ^D ...]
      % cat /tmp/log
      0: 0
      1: 1
      2: 0
      

      The possible attached patch seems to fix the problem reported by Martin.

    7. vstinner commented on Oct 22, 2019

      @vstinner
      Member

      I'm quite that this issue has been fixed by bpo-30225 which is part Python 3.7.4, Python 3.8.0 and newer. I mark the issue as duplicate of bpo-30225. Please reopen the bug if these versions still fail on NetBSD.

      commit 1c4670e
      Author: Victor Stinner <[email protected]>
      Date: Thu May 4 00:45:56 2017 +0200

      bpo-30225: Fix is_valid_fd() on macOS Tiger (bpo-1443)
      
      is_valid_fd() now uses fstat() instead of dup() on macOS to return 0
      on a pipe when the other side of the pipe is closed. fstat() fails
      with EBADF in that case, whereas dup() succeed.
      
    8. changed the title [-]Build does not work with closed stdin[/-] [+]Build does not work with closed stdin on NetBSD[/+] on Oct 22, 2019
    9. leot commented on Oct 24, 2019

      leotmannequin
      Mannequin

      Hello Victor,
      I can confirm that I can no longer reproduce this problem with Python
      3.7.5 on NetBSD and according a double-check to
      Python/pylifecycle.c:is_valid_fd() now fstat() is always used on
      NetBSD.

      Thanks for the update and for looking at it!

    10. vstinner commented on Oct 24, 2019

      @vstinner
      Member

      I can confirm that I can no longer reproduce this problem with Python
      3.7.5 on NetBSD

      Great!

    11. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      buildThe build process and cross-buildtype-featureA feature request or enhancement

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions