socket.recv(size, MSG_TRUNC) returns more than size bytes

@tiran

BPO	24933
Nosy	@tiran, @benjaminp, @berkerpeksag, @vadmium

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-08-25.11:28:15.516>
labels = ['extension-modules', 'type-bug', '3.7']
title = 'socket.recv(size, MSG_TRUNC) returns more than size bytes'
updated_at = <Date 2016-09-26.15:31:46.213>
user = 'https://bugs.python.org/AndreyWagin'

bugs.python.org fields:

activity = <Date 2016-09-26.15:31:46.213>
actor = 'christian.heimes'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2015-08-25.11:28:15.516>
creator = 'Andrey Wagin'
dependencies = []
files = []
hgrepos = []
issue_num = 24933
keywords = []
message_count = 7.0
messages = ['249114', '249121', '249127', '249214', '264343', '277427', '277429']
nosy_count = 5.0
nosy_names = ['christian.heimes', 'benjamin.peterson', 'berker.peksag', 'martin.panter', 'Andrey Wagin']
pr_nums = []
priority = 'high'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24933'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

In [1]: import socket

In [2]: sks = socket.socketpair(socket.AF_UNIX, socket.SOCK_DGRAM)

In [3]: sks[1].send("asdfasdfsadfasdfsdfsadfsdfasdfsdfasdfsadfa")
Out[3]: 42

In [4]: sks[0].recv(1, socket.MSG_PEEK | socket.MSG_TRUNC)
Out[4]: 'a\\x00\\x00\\x00\\xc0\\xbf8\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'

recv() returns a buffer. The size of this buffer is equal to the size of transferred data, but only the first symbol was initialized. What is the idea of this behavior.

Usually recv(sk, NULL, 0, socket.MSG_PEEK | socket.MSG_TRUNC) is used to get a message size. What is the right way to get a message size in Python?

sendto(4, "asdfasdfsadfasdfsdfsadfsdfasdfsd"..., 42, 0, NULL, 0) = 42
recvfrom(3, "a\\0n\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\5\\0\\0\\0\\0\\0\\0\\0\\2\\0\\0\\0"..., 1, MSG_TRUNC, NULL, NULL) = 42

I think the exit code is interpreted incorrectly. In this case it isn't equal to the number of bytes received. Then python copies this number of bytes from the buffer with smaller size, so it may access memory which are not allocated or allocated by someone else.

valgrind detects this type of errors:
[avagin@localhost ~]$ cat sock.py
import socket, os, sys

sks = socket.socketpair(socket.AF_UNIX, socket.SOCK_DGRAM)
pid = os.fork()
if pid == 0:
	sks[1].send("\0" * 4096)
	sys.exit(0)
sk = sks[0]
print sk.recv(1, socket.MSG_TRUNC )

[avagin@localhost \~]$ valgrind python sock.py
==25511== Memcheck, a memory error detector
==25511== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==25511== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==25511== Command: python sock.py
==25511== 
==25511== Syscall param write(buf) points to uninitialised byte(s)
==25511==    at 0x320B4F0940: \_\_write_nocancel (in /usr/lib64/libc-2.20.so)
==25511==    by 0x320B478D2C: \_IO_file_write@@GLIBC_2.2.5 (in /usr/lib64/libc-2.20.so)
==25511==    by 0x320B4794EE: \_IO_file_xsputn@@GLIBC_2.2.5 (in /usr/lib64/libc-2.20.so)
==25511==    by 0x320B46EE68: fwrite (in /usr/lib64/libc-2.20.so)
==25511==    by 0x369CC90210: ??? (in /usr/lib64/libpython2.7.so.1.0)
==25511==    by 0x369CC85EAE: ??? (in /usr/lib64/libpython2.7.so.1.0)
==25511==    by 0x369CC681AB: PyFile_WriteObject (in /usr/lib64/libpython2.7.so.1.0)
==25511==    by 0x369CCE08F9: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0)
==25511==    by 0x369CCE340F: PyEval_EvalCodeEx (in /usr/lib64/libpython2.7.so.1.0)
==25511==    by 0x369CCE3508: PyEval_EvalCode (in /usr/lib64/libpython2.7.so.1.0)
==25511==    by 0x369CCFC91E: ??? (in /usr/lib64/libpython2.7.so.1.0)
==25511==    by 0x369CCFDB41: PyRun_FileExFlags (in /usr/lib64/libpython2.7.so.1.0)

Evidently, the recv code doesn't know anything about MSG_TRUNC, which causes it to do incorrect things when the output length is greater than the buffer length.

There is the same behavior for python 3.4
>>> sks[1].send(b"asdfasdfsadfasdfsdfsadfsdfasdfsdfasdfsadfa")
42
>>> sks[0].recv(1, socket.MSG_PEEK | socket.MSG_TRUNC)
b'a\x00Nx\x94\x7f\x00\x00sadfasdfsdfsadfsdfasdfsdfasdfsadfa'
>>>

As far as I know, passing MSG_TRUNC into recv() is Linux-specific. I guess the “right” portable way to get a message size is to know it in advance, or guess and expand the buffer if MSG_PEEK cannot return the whole message.

Andrey: I don’t think we are accessing _unallocated_ memory (which could crash Python). If you look at _PyBytes_Resize(), I think it correctly allocates the memory, and just leaves it uninitialized.

Some options:

Document that arbitrary flags like Linux’s MSG_TRUNC not supported
Limit the returned buffer to the original buffer size
Raise an exception or warning if recv() returns more than the original buffer size
Reject unsupported flags like MSG_TRUNC
Initialize the expanded buffer with zeros

MSG_TRUNC literally causes a buffer overflow. In the example sock_recv() and friends only allocate a buffer of size 1 on the heap. With MSG_TRUNC recv() ignores the maximum size and writes beyond the buffer. We cannot recover from a buffer overflow because the overflow might have damanged other data structures. Instead Python should detect the problem and forcefully abort() the process with Py_FatalError().

Ah, I misunderstood MSG_TRUNC. It's not a buffer overflow. MSG_TRUNC does not write beyond the end of the buffer. In this example the libc function recv() writes two bytes into the buffer but returns a larger value than 2.

---

import socket
a, b = socket.socketpair(socket.AF_UNIX, socket.SOCK_DGRAM)
a.send(b'abcdefgh')
result = b.recv(2, socket.MSG_TRUNC)
print(len(result), result)

stdout: 2 b'ab'

To fix the wrong result of recv() with MSG_TRUNC, only resize when outlen < recvlen (line 3089).

To get the size of the message, you have to use recv_into() with a buffer.

---

a, b = socket.socketpair(socket.AF_UNIX, socket.SOCK_DGRAM)
a.send(b'abcdefgh')
msg = bytearray(2)
result = b.recv_into(msg, flags=socket.MSG_TRUNC)
print(result, msg)

---
stdout: 8 bytearray(b'ab')

@tiran

FWIW, PyPy just got the same bug report: https://foss.heptapod.net/pypy/pypy/-/issues/3864

We'll likely fix it in the way that @tiran suggested, "only resize when outlen < recvlen"

Since 3.15.1 (c38a347, backport of gh-157226), recv(size, MSG_TRUNC) and recvfrom(size, MSG_TRUNC) no longer return more than size bytes; instead, they raise ValueError: size larger than allocated size. Without MSG_PEEK, the datagram is consumed by the time the exception is raised, so the data is lost:

import socket
a, b = socket.socketpair(socket.AF_UNIX, socket.SOCK_DGRAM)
a.send(b'abcdefgh')
b.recv(2, socket.MSG_TRUNC)  # 3.15.0: b'ab\x00\x00\x00\x00\x00\x00'; 3.15.1: ValueError

Reported as a regression in gh-160622.

Navigation Menu

Repository navigation

socket.recv(size, MSG_TRUNC) returns more than size bytes #69121

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

socket.recv(size, MSG_TRUNC) returns more than size bytes #69121

Description

Activity

AndreyWagin commented on Aug 25, 2015

AndreyWagin commented on Aug 25, 2015

benjaminp commented on Aug 25, 2015

AndreyWagin commented on Aug 26, 2015

vadmium commented on Apr 27, 2016

tiran commented on Sep 26, 2016

tiran commented on Sep 26, 2016

cfbolz commented on Dec 3, 2022

kolyshkin commented on Oct 24, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions