Repository navigation
Piping the results of mypy to head can result in a broken pipe error #2893
Description
I have been working on getting mypy to return some meaningful results with a large codebase, and this requires writing a very large number of stubs. As a result, I see hundreds of lines of errors for very small files. To wade through all of this, I have been piping the results to head and gradually fixing various issues.
Every now and then, I have noticed that piping the results of the mypy output to head results in a broken pipe error.
Sometimes I see this error:
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe
At other times, I see this:
Traceback (most recent call last):
File "/home/.../bin/mypy", line 6, in <module>
main(__file__)
File "/home/.../lib/python3.5/site-packages/mypy/main.py", line 53, in main
f.write(m + '\n')
BrokenPipeError: [Errno 32] Broken pipe
Sometimes, I don't see any errors.
I can't provide a link to the project, due to copyright reasons, but you should be able to repeat this bug with any file which produces sufficiently long output if the results are fed to head, say with mypy some/file.py | head -n 10, etc.
Activity
[-]Piping the results of mypy to head can result in a Broken pipe error[/-][+]Piping the results of mypy to head can result in a broken pipe error[/+]refi64 commentedon Feb 22, 2017
I have a fix, but my mypy git repo is on an external hard disk that just accidentally got wiped...
FWIW putting this in
mypy/main.pyshould fix it:JukkaL commentedon Feb 24, 2017
I was able to reproduce this on macOS. Thanks for reporting this! A PR to fix this would be most welcome as well :-)
pkch commentedon Mar 1, 2017
This is not a
mypyproblem, but rather a well-established behavior of python interpreter.When
headgets its 10 lines, it knows it's done, so it closes the pipe. As a result, the OS (Linux/OSX but not Windows) sendsSIGPIPEsignal to the previous process in the shell pipe. The default (and usually correct) handling ofSIGPIPEis to terminate. This mechanism is designed to make shell pipes likecat hugefile | headwork efficiently - without having to read the entire file when you only need the first few lines.Python interpreter (for reasons that I don't understand) overrides the default handling of
SIGPIPE; specifically, it ignores this signal. As a result, any time the pipe is closed on the other end, instead of terminating, a python program would (by default) keep trying to write to the non-existent pipe, thus eventually causing a write error (how soon this happens depends on buffer sizes and policy).You can observe this behavior by simply running
python -c "print('\n'.join('a'*1000000))" | head. It would print the 10 lines, but then report aBrokenPipeErrorinstead of quietly exiting.To fix this, one would need to override python's handler with the default handler of
SIGPIPE(which terminates the process). As @kirbyfan64 suggested, it can be done withsignal.signal(signal.SIGPIPE, signal.SIG_DFL).Windows does not have a
SIGPIPEsignal at all, and so this solution wouldn't work. Instead, I suppose one could simply catchBrokenPipeErrorand terminate with exit code 0. But this is only a partial solution because some IO functions in python standard libraries catch and ignore this exception (instead of terminating). This is why sometimes the user sees a message likeException ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>-- and after that, some effort is wasted by the program that keeps producing output that won't be needed any more. I don't know of any feasible comprehensive solution to this problem under Windows.I wouldn't consider this a major bug, since the error message only happens after the desired number of lines are displayed by
head, so it's just a minor annoyance (unless someone uses the output in some kind of an automated system; or unless the output takes too long to generate).w0rp commentedon Mar 1, 2017
The problem is that
mypydoesn't handle the well defined behaviour in a pleasing manner, and it's not hard to solve. The program ought to be able to safely handle the pipe being closed early.w0rp commentedon Mar 1, 2017
I'm personally okay with a solution that works well in Linux, and doesn't work in Windows without breaking anything else in Windows. It's still better than it is now.
gvanrossum commentedon Mar 1, 2017
pkch commentedon Mar 1, 2017
Of course, I forgot:
SIGPIPEis sent when writing to any closed pipe, including network. This one reason is sufficient for python interpreter to not be ok with the default behavior.A program can safely accept default
SIGPIPEhandler only if it writes exclusively tostdoutand if it can assume that programs that follow it in the pipe also behave correctly (i.e, don't accept the default handler if they write, say, to a network).But really
SIGPIPEwas invented when it was hard to handle exceptions. In python, anything that can be done withSIGPIPEcan be done more flexibly and more portably (Windows!) with regular exceptions. So why not just catchBrokenPipeError?I know of only a couple technical problems with catching
BrokenPipeError:stdout; in general, it may be troublesome, but it should be easy formypysince I assume it only writes tostdout- just catch it in themainfunction;stdoutitself is being destroyed by python, exceptions are ignored but an unsuppressable error message is printed to the screen (issue11380); hopefully this is a minor problem.BrokenPipeError(defined in python as whenever the a socket sets errnoEPIPEorESHUTDOWN) doesn't quite work correctly (or as well asSIGPIPE), but I'm not aware of any specific cases of thatgvanrossum commentedon Mar 1, 2017
Catch BrokenPipeError when writing error messages (#2940)