Repository navigation
Ctypes callbacks shows problem on Windows Python (64bit) #61512
Description
| BPO | 17310 |
|---|---|
| Nosy | @vsajip, @amauryfa, @meadori |
| Superseder | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2016-08-03.14:27:39.924>
created_at = <Date 2013-02-27.15:51:14.123>
labels = ['ctypes', 'type-bug']
title = 'Ctypes callbacks shows problem on Windows Python (64bit)'
updated_at = <Date 2016-08-03.14:27:39.922>
user = 'https://bugs.python.org/MattClarke'bugs.python.org fields:
activity = <Date 2016-08-03.14:27:39.922>
actor = 'vinay.sajip'
assignee = 'none'
closed = True
closed_date = <Date 2016-08-03.14:27:39.924>
closer = 'vinay.sajip'
components = ['ctypes']
creation = <Date 2013-02-27.15:51:14.123>
creator = 'Matt.Clarke'
dependencies = []
files = ['29265']
hgrepos = []
issue_num = 17310
keywords = []
message_count = 12.0
messages = ['183156', '184078', '184444', '184457', '184460', '187650', '245146', '245148', '245151', '270900', '270917', '271906']
nosy_count = 5.0
nosy_names = ['vinay.sajip', 'amaury.forgeotdarc', 'meador.inge', 'Matt.Clarke', 'Patrick Stewart']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '20160'
type = 'behavior'
url = 'https://bugs.python.org/issue17310'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6']
Activity
mattclarke commentedon Feb 27, 2013
I have had an issue arise with ctypes callbacks with 64bit Python on Windows.
Note: everything works fine with 32bit Python on Windows and on 32bit and 64bit Linux.
I have created a simple example to illustrate the issue I have (see attachment), but the real-life issue occurs with using Python to interact with the EPICS control software (http://www.aps.anl.gov/epics/) used at many major scientific institutes.
Basically, if I have a C callback that takes a struct (by value) greater than 8 bytes then the callback returns nonsense. 8 bytes or less works fine.
Stepping through with the Windows debugger, if appears that something goes amiss between the callback being called in C and the closure_fcn(ffi_cif *cif, void *resp, void **args, void *userdata) function in ctypes's callback.c file. Unfortunately, the debugger won't let me step in between those two points.
Looking at the memory I can see the original data in memory at some memory address, X, and a copy of the data at X+40 bytes, but the args in the closure_fcn points at X-40 bytes (which is junk).
Using 32bit Python the data copy is at X-40 bytes and the args pointer in the closure_fcn also points at this.
EPICS has some 64bit C/C++ clients that work fine using callbacks on Windows. Likewise, doing the same sort of thing as ctypes does with EPICS from C# using PInvoke works fine.
Any help would be much appreciated.
amauryfa commentedon Mar 13, 2013
Is "_pack_ = 1" correct? Did you compile your C library with /Zp1 or similar?
Also check that ctypes.sizeof(callback_t) matches the one given by the C compiler.
mattclarke commentedon Mar 18, 2013
Hi Amaury.
I have tried removing pack and using /Zp1, but it makes no difference.
The size of callback_t and the one in C are 8 bytes.
Thanks,
Matt
On 13 March 2013 13:19, Amaury Forgeot d'Arc <[email protected]> wrote:
amauryfa commentedon Mar 18, 2013
Sorry, I asked the wrong question; callback_t is a function pointer, so 8 bytes is expected.
What is sizeof(myst_args) both in C and Python?
mattclarke commentedon Mar 18, 2013
Hi Amaury.
They are both 12 bytes.
Matt
mattclarke commentedon Apr 23, 2013
It seems that any argument greater than 8 bytes is automatically converted
to a references. Thus, changing to using ctypes.POINTER works. For example:
Quite a simple solution in the end. Is it worth documenting this on the
ctypes page?
Thanks for your help,
Matt
On 18 March 2013 15:19, Matt Clarke <[email protected]> wrote:
vsajip commentedon Jun 10, 2015
Although Matt was able to work around his problem, this problem seems to still be present in 2.7 as well as all 3.x versions. I think I've found the cause of the problem: in 64-bit code, the calling conventions for passing structures by value are different. From this page:
https://msdn.microsoft.com/en-us/library/zthk2dkh(v=vs.90).aspx
I would point to "Structs/unions of size 8, 16, 32, or 64 bits and __m64 will be passed as if they were integers of the same size. Structs/unions other than these sizes will be passed as a pointer to memory allocated by the caller. For these aggregate types passed as a pointer (including __m128), the caller-allocated temporary memory will be 16-byte aligned."
The code in ffi_prep_incoming_args_SYSV (see https://hg.python.org/cpython/file/a1b76c1c3be8/Modules/_ctypes/libffi_msvc/ffi.c#l368) assumes (see lines 399, 402) that all value parameters are always passed inline on the stack - which is plainly not as per the documentation I linked to, for 64-bit code.
vsajip commentedon Jun 10, 2015
I note that the code for CFFI has a fix in ffi_prep_incoming_args_SYSV, as follows:
(Source: https://bitbucket.org/cffi/cffi/src/abc8ff5b2885b3d9f22bbb314a011b8dd63c9e14/c/libffi_msvc/ffi.c?at=default)
vsajip commentedon Jun 10, 2015
I can confirm that the CFFI patch works as expected on Python 2.7.10.
PatrickStewart commentedon Jul 21, 2016
This also fixes python 3.5
PatrickStewart commentedon Jul 21, 2016
I've attached a patch with an extra fix to the duplicated bpo-20160 http://bugs.python.org/issue20160
vsajip commentedon Aug 3, 2016
I'm closing this as a duplicate of bpo-20160 - it appears to be the same bug.