Repository navigation
Expose new copy_file_range() syscall in os module. #71013
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2020-04-22.16:15:30.897>
created_at = <Date 2016-04-22.11:40:39.381>
labels = ['type-feature', 'library']
title = 'Expose new copy_file_range() syscall in os module.'
updated_at = <Date 2020-04-22.16:15:30.896>
user = 'https://bugs.python.org/StyXman'bugs.python.org fields:
activity = <Date 2020-04-22.16:15:30.896>
actor = 'benjamin.peterson'
assignee = 'none'
closed = True
closed_date = <Date 2020-04-22.16:15:30.897>
closer = 'benjamin.peterson'
components = ['Library (Lib)']
creation = <Date 2016-04-22.11:40:39.381>
creator = 'StyXman'
dependencies = []
files = ['42619', '42628', '42638', '42640', '42651', '42708', '42829', '43310', '43318', '43319', '43624', '43639', '43640', '43688']
hgrepos = []
issue_num = 26826
keywords = ['patch']
message_count = 54.0
messages = ['264000', '264001', '264004', '264013', '264067', '264071', '264072', '264087', '264166', '264167', '264302', '264304', '264320', '264322', '264336', '264362', '264365', '264371', '264429', '264430', '264433', '264457', '264494', '264502', '264537', '264797', '265121', '265405', '265441', '267899', '268011', '268015', '268019', '269810', '269825', '269881', '269882', '270171', '274746', '317365', '317366', '317667', '317679', '337694', '344100', '344576', '344580', '344582', '344584', '344586', '344602', '344630', '344649', '344672']
nosy_count = 10.0
nosy_names = ['facundobatista', 'ncoghlan', 'vstinner', 'giampaolo.rodola', 'StyXman', 'petr.viktorin', 'neologix', 'martin.panter', 'desbma', 'pablogsal']
pr_nums = ['7255']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26826'
versions = ['Python 3.6']
Activity
StyXman commentedon Apr 22, 2016
copy_file_range() has been introduced in the Linux kernel since version 4.5 (mid march 2016). This new syscall allows to copy data from one fd to another without passing by user space, improving speed in most cases. You can read more about it here:
https://lwn.net/Articles/659523/
I intend to start working on adding a binding for it in the os module and then, if it's available, use it in shutils.copy() to improve its efficiency. I have a couple of questions: If the syscall is not available, should I implement a user space alternative or should the method not exist at all?
tiran commentedon Apr 22, 2016
Thanks, looks interesting.
We usually wait until syscalls are generally available in common distros and have bindings in glibc. It makes it easier to test the feature.
vstinner commentedon Apr 22, 2016
Usually, yeah. os.urandom() uses syscall() to use the new getrandom() of Linux since it's still not exposed in the GNU libc...
https://sourceware.org/bugzilla/show_bug.cgi?id=17252
Status: NEW
vadmium commentedon Apr 22, 2016
Tangentially related: bpo-25156, about using sendfile() to copy files in shutil.
StyXman commentedon Apr 23, 2016
Debian Sid, arch Linux and Fedora FC24 (alpha) already have linux-4.5, others would certainly follow soon. Meanwhile, I could start developing the patch and we could review it when you think it's appropriate.
vstinner commentedon Apr 23, 2016
Kerbel support ok, but what about the libc support? Do you know if it is
planned? Or do you want to use the syscall() low-level API. If we take the
syscall() path, I suggest to make the function private in the os module.
Wait until the API is standardized in the libc. Sometimes the libc changes
minor things, ir's not always a thin wrapper to the syscall.
StyXman commentedon Apr 23, 2016
Already there:
http://man7.org/linux/man-pages/man2/copy_file_range.2.html
vstinner commentedon Apr 24, 2016
That's the manual page of the Linux kernel, not the glibc. It doesn't
mean that the glibc implemented it.
StyXman commentedon Apr 25, 2016
Then I don't know. My only worry is whether having the method local to os would allow shutils.copy() to use it or not.
I will start by looking at other similar functions there to see to do it. urandom() looks like a good starting point.
vstinner commentedon Apr 25, 2016
If we add a new private function to the os module (ex:
os._copy_file_range), it can be used in shutil if available. But it
would be a temporary solution until we declare the API stable. Maybe
it's ok to add the API as public today.
StyXman commentedon Apr 26, 2016
Ok, I have a preliminary version of the patch. It has several parts:
Several points:
For the time being, flags must be 0, so I was not sure whether put the argument or not. Just in case, I put it.
I'm not sure how to test for availability, so configure defines HAVE_COPY_FILE_RANGE.
No tests yet.
Talking about tests, I tried copying a 325MiB on an SSD, f2fs. Here are the times:
Old user space copy:
New copy_file_range:
Some 20% improvement, but notice that the buffer size is 1024 times Python's default size (16MiB vs. 16KiB).
One difference that I notice in semantics is that if the file is not open in binary form, but the file is binary, you get no UnicodeDecodeError (because the data never reaches userspace).
Let me know what you think.
StyXman commentedon Apr 26, 2016
Version without the NEWS and ACKS change.
StyXman commentedon Apr 26, 2016
Hmm, I just noticed that it doesn't fallback to normal copy if the arguments are not valid (EBADF, EXDEV). Back to the drawing board...
30 remaining items
StyXman commentedon May 25, 2018
I'm really sorry, but I don't have time to continue with this (new daughter!). Can someone else pick it up?
giampaolo commentedon May 25, 2018
Check for availability in configure.ac appears to be broken.
giampaolo commentedon Mar 11, 2019
Little update about this. According to:
http://man7.org/linux/man-pages/man2/copy_file_range.2.html
...it seems glibc 2.27 released on 2018-02-01 includes copy_file_range(). I'm not the best candidate for giving advice on C syscalls definitions/availability, but FWIW this works on my Linux 4.15:
I think (but not sure) this is supposed to fix this condition, which appears to be the major blocker here:
https://github.com/python/cpython/blob/9a177061cd7190eabf40efd31e8981e0bccd5dc4/Lib/test/test_os.py#L258-L261
pablogsal commentedon May 31, 2019
New changeset aac4d03 by Pablo Galindo in branch 'master':
bpo-26826: Expose copy_file_range in the os module (GH-7255)
aac4d03
vstinner commentedon Jun 4, 2019
shutil copy functions would definitively benefit of using copy_file_range() if available. Can someone please open a separated issue for shutil?
giampaolo commentedon Jun 4, 2019
shutil.copyfile() already uses sendfile() which basically provides the same performances. sendfile() should be preferred though because it’s supported since Linux 2.6.33.
pablogsal commentedon Jun 4, 2019
But copy_file_rane can leverage more filesystem features like deduplication and copy offload stuff.
vstinner commentedon Jun 4, 2019
Giampaolo Rodola':
Pablo Galindo Salgado:
We can use copy_file_range() if available, or fallback to sendfile().
giampaolo commentedon Jun 4, 2019
I think data deduplication / CoW / reflink copy is better implemented via FICLONE. "cp --reflink" uses it, I presume because it's older than copy_file_range(). I have a working patch adding CoW copy support for Linux and OSX (but not Windows). I think that should be exposed as a separate shutil.reflink() though, and copyfile() should just do a standard copy.
giampaolo commentedon Jun 4, 2019
Actually "man copy_file_range" claims it can do server-side copy, meaning no network traffic between client and server if *src* and *dst* live on the same network fs. So I agree copy_file_range() should be preferred over sendfile() after all. =)
I have a wrapper for copy_file_range() similar to what I did in shutil in bpo-33671 which I can easily integrate, but I wanted to land this one first:
https://bugs.python.org/issue37096
Also, I suppose we cannot land this in time for 3.8?
vstinner commentedon Jun 4, 2019
Please open a new issue to discuss how it can used in shutil ;-)
vstinner commentedon Jun 4, 2019
I created bpo-37157: "shutil: add reflink=False to file copy functions to control clone/CoW copies (use copy_file_range)".
--
The new os.copy_file_range() should be documented at:
https://docs.python.org/dev/whatsnew/3.8.html#os
giampaolo commentedon Jun 5, 2019
Use copy_file_range() in shutil.copyfile():
https://bugs.python.org/issue37159