test_ctypes fails when building python with clang

@ronaldoussoren

BPO	13370
Nosy	@ronaldoussoren, @ned-deily, @meadori
Files	issue_13370.txt issue_13370-2.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ronaldoussoren'
closed_at = <Date 2012-08-29.09:43:26.121>
created_at = <Date 2011-11-08.14:11:14.294>
labels = ['ctypes', 'type-bug']
title = 'test_ctypes fails when building python with clang'
updated_at = <Date 2012-08-29.09:43:26.120>
user = 'https://github.com/ronaldoussoren'

bugs.python.org fields:

activity = <Date 2012-08-29.09:43:26.120>
actor = 'ronaldoussoren'
assignee = 'ronaldoussoren'
closed = True
closed_date = <Date 2012-08-29.09:43:26.121>
closer = 'ronaldoussoren'
components = ['ctypes']
creation = <Date 2011-11-08.14:11:14.294>
creator = 'ronaldoussoren'
dependencies = []
files = ['26821', '26984']
hgrepos = []
issue_num = 13370
keywords = ['patch', 'needs review']
message_count = 31.0
messages = ['147299', '147301', '147302', '152662', '152690', '152699', '152714', '152717', '152718', '152801', '152804', '152923', '155148', '156216', '168203', '168275', '168672', '168676', '168753', '168852', '168923', '168950', '169005', '169006', '169071', '169089', '169094', '169098', '169117', '169120', '169121']
nosy_count = 7.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'meador.inge', 'python-dev', 'ezberch', 'Anton.Korobeynikov', 'Stan.Seibert']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13370'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

test_ctypes fails when building and testing on OSX 10.7 (Lion).

This is likely caused by changes in the compiler w.r.t. the calling convention. I've worked around this problem in PyObjC, and will try to port that workaround to ctypes.

BTW. Using --with-system-ffi does not fix the issue for me

test_ctypes fails when building and testing on OSX 10.7 (Lion).

What is the error message?

I should have included the failures in my first message, sorry about that.

IIRC the workaround in PyObjC I mention in my initial message does explicit sign extension when needed.

W.r.t to the first couple of failures, not that:

struct.unpack('b', struct.pack('B', 130)) == (-126,)

======================================================================
FAIL: test_byte (ctypes.test.test_cfuncs.CFunctions)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Library/Frameworks/PythonDev.framework/Versions/2.7/lib/python2.7/ctypes/test/test_cfuncs.py", line 21, in test_byte
    self.assertEqual(self.S(), -126)
AssertionError: 130 != -126

======================================================================
FAIL: test_byte_plus (ctypes.test.test_cfuncs.CFunctions)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Library/Frameworks/PythonDev.framework/Versions/2.7/lib/python2.7/ctypes/test/test_cfuncs.py", line 27, in test_byte_plus
    self.assertEqual(self.S(), -126)
AssertionError: 130 != -126

======================================================================
FAIL: test_short (ctypes.test.test_cfuncs.CFunctions)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Library/Frameworks/PythonDev.framework/Versions/2.7/lib/python2.7/ctypes/test/test_cfuncs.py", line 45, in test_short
    self.assertEqual(self.S(), -32766)
AssertionError: 32770 != -32766

======================================================================
FAIL: test_short_plus (ctypes.test.test_cfuncs.CFunctions)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Library/Frameworks/PythonDev.framework/Versions/2.7/lib/python2.7/ctypes/test/test_cfuncs.py", line 51, in test_short_plus
    self.assertEqual(self.S(), -32766)
AssertionError: 32770 != -32766

======================================================================
FAIL: test_doubleresult (ctypes.test.test_functions.FunctionTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Library/Frameworks/PythonDev.framework/Versions/2.7/lib/python2.7/ctypes/test/test_functions.py", line 143, in test_doubleresult
    self.assertEqual(result, -21)
AssertionError: 65771.0 != -21

======================================================================
FAIL: test_floatresult (ctypes.test.test_functions.FunctionTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Library/Frameworks/PythonDev.framework/Versions/2.7/lib/python2.7/ctypes/test/test_functions.py", line 131, in test_floatresult
    self.assertEqual(result, -21)
AssertionError: 65771.0 != -21

======================================================================
FAIL: test_intresult (ctypes.test.test_functions.FunctionTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Library/Frameworks/PythonDev.framework/Versions/2.7/lib/python2.7/ctypes/test/test_functions.py", line 105, in test_intresult
    self.assertEqual(result, -21)
AssertionError: 65771 != -21

======================================================================
FAIL: test_longdoubleresult (ctypes.test.test_functions.FunctionTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Library/Frameworks/PythonDev.framework/Versions/2.7/lib/python2.7/ctypes/test/test_functions.py", line 155, in test_longdoubleresult
    self.assertEqual(result, -21)
AssertionError: 65771.0 != -21

I can't reproduce this on Lion using the 2.7 or default branches. I can't build with GCC at the moment because of bpo-13241. I am using the clang that ships with Lion:

motherbrain:cpython meadori$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.7.2
BuildVersion: 11C74
motherbrain:cpython meadori$ clang --version
Apple clang version 3.0 (tags/Apple/clang-211.12) (based on LLVM 3.0svn)
Target: x86_64-apple-darwin11.2.0
Thread model: posix

It looks like the test failures do not show up with debug enabled (--with-pydebug). I'm able to reproduce them with:

./configure MACOSX_DEPLOYMENT_TARGET=10.7 CC=clang

I can reproduce it without --with-pydebug. Thanks. I will investigate.

The reason for this issue appears to be a bug in the clang optimizer. This is why it only shows up with debug disabled. When I compile with -O0 instead of -O3, the test passes.

This issue from the clang tracker may be the same issue:
http://llvm.org/bugs/show_bug.cgi?id=9524

So I have debugged the first failure and it has to do with the sign extending of 'signed char' parameters. The first test case basically does this:

   _dll = CDLL(_ctypes_test.__file__)
   _dll.tf_b.restype = c_byte
   _dll.tf_b.argtypes = (c_byte,)
   _dll.tf_b(-126)

where 'tf_b' is defined in C code like:

signed char tf_b(signed char c) { return c/3; }

Clang with -O3 generates code for 'tf_b' like:

0x0000000100000ed0 <tf_b+0>: push %rbp
0x0000000100000ed1 <tf_b+1>: mov %rsp,%rbp
0x0000000100000ed4 <tf_b+4>: movslq %edi,%rax
0x0000000100000ed7 <tf_b+7>: imul $0x55555556,%rax,%rax
0x0000000100000ede <tf_b+14>: mov %rax,%rcx
0x0000000100000ee1 <tf_b+17>: shr $0x3f,%rcx
0x0000000100000ee5 <tf_b+21>: shr $0x20,%rax
0x0000000100000ee9 <tf_b+25>: add %ecx,%eax
0x0000000100000eeb <tf_b+27>: movsbl %al,%eax
0x0000000100000eee <tf_b+30>: pop %rbp
0x0000000100000eef <tf_b+31>: retq

See how 'movslq' is used to sign extend the first argument? Since the first argument is a 'signed char', that should be 'movsbq'. I am pretty sure this is a clang bug.

I am going to see if this is fixed in later versions of clang. I will also search the clang tracker to see if this has been reported.

Since the first argument is a 'signed char', that should be 'movsbq'.

Hmmm, on second thought I am not 100% sure about that exact statement, but I still think this is most likely a clang bug having something to do with sign extension.

It turns out that this isn't OS X specific. I can reproduce the exact same issue by building with clang on a 64-bit Fedora 16 box.

The patch as it stands causes _ctypes build failures when using the 10.4u SDK as in the traditional 10.3+ 32-bit-only installer configuration. Apparently, in ppc-ffi_darwin.c, the skip that the patch removes was because OSCacheControl.h doesn't exist in the 10.4u SDK. It is in the 10.5 SDK and _ctypes.so builds and tests OK on both PPC and i386 with it.

--- a/Modules/_ctypes/libffi_osx/powerpc/ppc-ffi_darwin.c
+++ b/Modules/_ctypes/libffi_osx/powerpc/ppc-ffi_darwin.c
@@ -37,19 +37,10 @@
 #include <ppc-darwin.h>
 #include <architecture/ppc/mode_independent_asm.h>
 
-#if 0
 #if defined(POWERPC_DARWIN)
 #include <libkern/OSCacheControl.h>	// for sys_icache_invalidate()
 #endif
 
-#else
-
-#pragma weak sys_icache_invalidate
-extern void sys_icache_invalidate(void *start, size_t len);
-
-#endif

./Modules/_ctypes/libffi_osx/powerpc/ppc-ffi_darwin.c:41:67: error: libkern/OSCacheControl.h: No such file or directory

The change to ppc-ffi_darwin.c was unintentional and should be removed. I must have been sleeping during the merge, as I only intended to copy the files for i386/x86_64 and not for PPC as the latter architecture didn't have problems when compiled with clang.

I've copied the ppc-ffi_darwin.c file from Python's tree into the PyObjC tree to avoid problems when there are future merges.

I just noticed another anomaly in the patch. In x86-ffi_darwin.c there appears to be some bogus code duplicating ffi_prep_args. Could you review and clean up the patch before we apply it?

I didn't have time to look into this today and will look into this tomorrow.

What code duplication do you mean?

The code in the switch statement at line 71 looks similar for the various case, but it isn't: there is variation in the types uses in the casts (and particularly in the cast used to read the input value).

Attached version of the patch without the change to the powerpc specific code.

I will commit this version later today (after another testing round).

Sorry, I meant this part of the patch:

diff --git a/Modules/_ctypes/libffi_osx/x86/x86-ffi_darwin.c b/Modules/_ctypes/libffi_osx/
--- a/Modules/_ctypes/libffi_osx/x86/x86-ffi_darwin.c
+++ b/Modules/_ctypes/libffi_osx/x86/x86-ffi_darwin.c
@@ -35,6 +35,8 @@
 /* ffi_prep_args is called by the assembly routine once stack space
  has been allocated for the function's arguments */
 
+void ffi_prep_args(char *stack, extended_cif *ecif);
+
 void ffi_prep_args(char *stack, extended_cif *ecif)
 {
     register unsigned int i;

That's not code duplication, but a way to silence a compiler warning. I added it while trying to get pyobjc to compile without any warnings.

I noticed earlier today that test_ctypes passes for Python 3.3 both with and without my patch. This is on OSX 10.8.1 with an up-to-date Xcode.

Configure arguments:

'--enable-framework' '--with-pydebug' '--enable-universalsdk=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk' '--with-universal-archs=intel' '--with-framework-name=PythonDev' 'MACOSX_DEPLOYMENT_TARGET=10.8' 'CC=clang' 'CXX=clang++'

That's why I haven't commited the patch yet. I'm installing Xcode in a OSX 10.7 VM to check if the patch is necessary on that platform.

Bah, I just noticed that you built --with-pydebug enabled. That hides the problem. I was fooled myself there. But I've since verified that the failure still exists when built on 10.7 with Xcode 4.4.1 so the patch should go in.

Duh...

3.3 without patch fails when I leave out '--with-pydebug'. I'm now running the testsuite for 2.7 and 3.2 with patch and will commit once those are ready, which should be within an hour from now.

New changeset 58a87c02692c by Ronald Oussoren in branch '2.7':
Fix bpo-13370: Ensure that ctypes works on Mac OS X when Python is compiled using the clang compiler
http://hg.python.org/cpython/rev/58a87c02692c

New changeset a425f2697273 by Ronald Oussoren in branch '3.2':
Fix bpo-13370: Ensure that ctypes works on Mac OS X when Python is compiled using the clang compiler
http://hg.python.org/cpython/rev/a425f2697273

New changeset bed8dd30309d by Ronald Oussoren in branch 'default':
Fix bpo-13370: Ensure that ctypes works on Mac OS X when Python is
http://hg.python.org/cpython/rev/bed8dd30309d

The issue should be gone in the upcoming 3.3rc1 release and the next stable releases of 3.2 and 2.7.

Navigation Menu

Repository navigation

test_ctypes fails when building python with clang #57579

9 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

test_ctypes fails when building python with clang #57579

Description

Activity

ronaldoussoren commented on Nov 8, 2011

vstinner commented on Nov 8, 2011

ronaldoussoren commented on Nov 8, 2011

meadori commented on Feb 5, 2012

ned-deily commented on Feb 5, 2012

meadori commented on Feb 5, 2012

ezberch commented on Feb 6, 2012

meadori commented on Feb 6, 2012

meadori commented on Feb 6, 2012

meadori commented on Feb 7, 2012

9 remaining items

ned-deily commented on Aug 21, 2012

ronaldoussoren commented on Aug 22, 2012

ned-deily commented on Aug 23, 2012

ronaldoussoren commented on Aug 23, 2012

ronaldoussoren commented on Aug 24, 2012

ronaldoussoren commented on Aug 24, 2012

ned-deily commented on Aug 24, 2012

ronaldoussoren commented on Aug 24, 2012

ronaldoussoren commented on Aug 24, 2012

ned-deily commented on Aug 24, 2012

ronaldoussoren commented on Aug 25, 2012

python-dev commented on Aug 25, 2012

ronaldoussoren commented on Aug 25, 2012

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions