Skip to content

python (x64) ctypes incorrectly pass structures parameter #56044

Closed
Closed
python (x64) ctypes incorrectly pass structures parameter#56044
Assignees
theller
Labels
topic-ctypestype-bugAn unexpected behavior, bug, or error

Description

@AbrahamSoedjito
AbrahamSoedjito
mannequin
opened on Apr 12, 2011
BPO 11835
Nosy @theller, @amauryfa, @abalkin, @mattip
Superseder
  • bpo-20160: broken ctypes calling convention on MSVC / 64-bit Windows (large structs)
    		•
    Files
  • test_issue11835.patch: from 2.7
  • issue11835_patch.diff: Patch for this issue
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/theller'
closed_at = <Date 2015-01-16.03:29:48.004>
created_at = <Date 2011-04-12.15:39:25.915>
labels = ['ctypes', 'type-bug']
title = 'python (x64) ctypes incorrectly pass structures parameter'
updated_at = <Date 2015-01-16.03:29:48.003>
user = 'https://bugs.python.org/AbrahamSoedjito'

    bugs.python.org fields:

    activity = <Date 2015-01-16.03:29:48.003>
actor = 'berker.peksag'
assignee = 'theller'
closed = True
closed_date = <Date 2015-01-16.03:29:48.004>
closer = 'berker.peksag'
components = ['ctypes']
creation = <Date 2011-04-12.15:39:25.915>
creator = 'Abraham.Soedjito'
dependencies = []
files = ['21633', '22899']
hgrepos = []
issue_num = 11835
keywords = ['patch']
message_count = 7.0
messages = ['133583', '133591', '133593', '133596', '142083', '142092', '234094']
nosy_count = 9.0
nosy_names = ['theller', 'amaury.forgeotdarc', 'belopolsky', 'cgohlke', 'santoso.wijaya', 'Abraham.Soedjito', 'jwhitecl', 'vladris', 'mattip']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '20160'
type = 'behavior'
url = 'https://bugs.python.org/issue11835'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2', 'Python 3.3']

    Activity

    1. AbrahamSoedjito commented on Apr 12, 2011

      @AbrahamSoedjito
      AbrahamSoedjitomannequin
      on Apr 12, 2011
      MannequinAuthor
      void __cdecl foo(unsigned __int32 a,
                 unsigned __int32 b,
                 unsigned __int32 c,
                 unsigned __int32 d,
                 unsigned __int32 e,
                 unsigned __int32 f,
                 unsigned __int32 g);

struct myStruct
{
   unsigned __int32 a;
   unsigned __int32 b;
   unsigned __int32 c;
   unsigned __int32 d;
   unsigned __int32 e;
   unsigned __int32 f;
   unsigned __int32 g;
}
void __cdecl bar(myStruct s);

void __cdecl errorPassingParameter(myStruct s1,
                                   myStruct s2,
                                   unsigned __int32 x);

      Calling foo and bar from python completed successfully, calling errorParsingParameter resulted in stack corruption. It seems that python passed an extra pointer in the stack for s2.

    2. AbrahamSoedjito
      added
      type-crashA hard crash of the interpreter, possibly with a core dump
      on Apr 12, 2011
    3. AbrahamSoedjito
      assigned
      theller
      on Apr 12, 2011
    4. AbrahamSoedjito
      added
      topic-ctypes
      on Apr 12, 2011

    5. santosowijaya commented on Apr 12, 2011

      @santosowijaya
      santosowijayamannequin
      on Apr 12, 2011
      Mannequin

      I can reproduce this with 2.7 and 3.2 64-bit builds on Windows:

      D:\Temp\cdll\x64\Release>C:\Python32\python.exe
Python 3.2 (r32:88445, Feb 20 2011, 21:30:00) [MSC v.1500 64 bit (AMD64)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import *
>>> libcdll = CDLL('cdll')
>>> libcdll.foo(1, 2, 3, 4, 5, 6, 7)
a: 1; b: 2; c: 3; d: 4; e: 5; f: 6; g: 7
0
>>> class MyStruct(Structure):
...     _fields_ = [
... ('a', c_int32),
... ('b', c_int32),
... ('c', c_int32),
... ('d', c_int32),
... ('e', c_int32),
... ('f', c_int32),
... ('g', c_int32),
... ]
...
>>> s1 = MyStruct(1, 2, 3, 4, 5, 6, 7)
>>> s2 = MyStruct(0, 9, 8, 7, 6, 5, 4)
>>> libcdll.bar(s2)
s.a: 0; s.b: 9; s.c: 8; s.d: 7; s.e: 6; s.f: 5; s.g: 4
0
>>> libcdll.errorPassingParameter(s1, s2, 42)
s1.a: 1; s1.b: 2; s1.c: 3; s1.d: 4; s1.e: 5; s1.f: 6; s1.g: 7
s2.a: 28; s2.b: 0; s2.c: 851972; s2.d: 0; s2.e: 31545776; s2.f: 0; s2.g: 0
x: 39805032
0
>>>

    6. santosowijaya commented on Apr 12, 2011

      @santosowijaya
      santosowijayamannequin
      on Apr 12, 2011
      Mannequin

      32-bit (2.6) is fine:

      D:\Temp\cdll\Release>C:\Python26\python.exe
Python 2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import *
>>> libcdll = CDLL('cdll')
>>> class MyStruct(Structure):
...     _fields_ = []
...     for name in ('a', 'b', 'c', 'd', 'e', 'f', 'g'):
...         _fields_.append((name, c_int32))
...
>>> MyStruct._fields_
[('a', <class 'ctypes.c_long'>), ('b', <class 'ctypes.c_long'>), ('c', <class 'c
types.c_long'>), ('d', <class 'ctypes.c_long'>), ('e', <class 'ctypes.c_long'>),
 ('f', <class 'ctypes.c_long'>), ('g', <class 'ctypes.c_long'>)]
>>> s1 = MyStruct(1, 2, 3, 4, 5, 6, 7)
>>> s2 = MyStruct(0, 9, 8, 7, 6, 5, 4)
>>> libcdll.errorPassingParameter(s1, s2, 42)
s1.a: 1; s1.b: 2; s1.c: 3; s1.d: 4; s1.e: 5; s1.f: 6; s1.g: 7
s2.a: 0; s2.b: 9; s2.c: 8; s2.d: 7; s2.e: 6; s2.f: 5; s2.g: 4
x: 42
0
>>>

    7. santosowijaya commented on Apr 12, 2011

      @santosowijaya
      santosowijayamannequin
      on Apr 12, 2011
      Mannequin

      Attaching a ctypes unittest (against 2.7 branch) that will expose this bug in regrtest.

    8. vladris commented on Aug 14, 2011

      @vladris
      vladrismannequin
      on Aug 14, 2011
      Mannequin

      Attached patch for this issue.

      This only happens on MSVC x64 (I actually tired to repro on Arch Linux x64 before starting work on it and it didn't repro).

      What happens is that MSVC on x64 always passes structures larger than 8 bytes by reference. See here: http://msdn.microsoft.com/en-us/library/ms235286(v=vs.90).aspx

      Now this was accounted for in callproc.c, line 1143 in development branch with this:

              if (atypes[i]->type == FFI_TYPE_STRUCT
#ifdef _WIN64
            && atypes[i]->size <= sizeof(void *)
#endif
            )
            avalues[i] = (void *)args[i].value.p;
        else
            avalues[i] = (void *)&args[i].value;

      This fix wasn't made in libffi_msvc/ffi.c though. Here, regardless of whether we have x64 or x86 build, if z >= sizeof(int) we will hit else branch in libffi_msvc/ffi.c at line 114 and do:

        else
{
  memcpy(argp, \*p_argv, z);
}
  p_argv++;
  argp += z;

      In our case, we copy 28 bytes as arguments (size of our structure) but in fact for x64 we only need 8 as structure is passed by reference so argument is just a pointer. My patch will adjust z before hitting if statement on x64 and it will cause correct copy as pointer.

    9. vladris commented on Aug 15, 2011

      @vladris
      vladrismannequin
      on Aug 15, 2011
      Mannequin

      Changing type to behavior as it doesn't crash on 3.3. I believe issue was opened against 2.6 and Santoso changed it to 2.7 and up where there is no crash.

      Another data point: there is similar fix in current version of libffi here: https://github.com/atgreen/libffi/blob/master/.pc/win64-struct-args/src/x86/ffi.c

      Since at the moment we are not integrating new libffi, I believe my fix should do (libffi fix is slightly different but I'm matching what we have in callproc.c which is not part of libffi).

    10. vladris
      added
      type-bugAn unexpected behavior, bug, or error
      and removed
      type-crashA hard crash of the interpreter, possibly with a core dump
      on Aug 15, 2011

    11. mattip commented on Jan 15, 2015

      @mattip
      mattip
      on Jan 15, 2015
      Contributor

      This is a duplicate of bpo-20160, and has been fixed. Can someone mark it as a duplicate and close it?

    12. berkerpeksag
      closed this as completedon Jan 16, 2015
    13. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    topic-ctypestype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions