Repository navigation
Build Failure on Windows, Python 3.6 GCC Compilers: UnicodeDecodeError. #16668
Description
updating to the latest master , an UnicodeDecodeError occurred even after execute git clean -xfd
Reproducing code example:
No code example.
Error message:
CCompilerOpt.cc_test_flags[989] : testing flags (/WX)
CCompilerOpt.__init__[1137] : feature 'AVX512_KNL' is disabled, MSVC compiler doesn't support it
CCompilerOpt.__init__[1137] : feature 'AVX512_KNM' is disabled, MSVC compiler doesn't support it
CCompilerOpt.__init__[1621] : check requested baseline
CCompilerOpt.cc_test_flags[989] : testing flags (/arch:SSE)
Traceback (most recent call last):
File "setup.py", line 499, in <module>
setup_package()
File "setup.py", line 491, in setup_package
setup(**metadata)
File "D:\Source\numpy\numpy\distutils\core.py", line 169, in setup
return old_setup(**new_attr)
File "C:\Python36\lib\site-packages\setuptools\__init__.py", line 143, in setup
return distutils.core.setup(**attrs)
File "C:\Python36\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "C:\Python36\lib\distutils\dist.py", line 955, in run_commands
self.run_command(cmd)
File "C:\Python36\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "D:\Source\numpy\numpy\distutils\command\build_ext.py", line 111, in run
build_clib.run()
File "D:\Source\numpy\numpy\distutils\command\build_clib.py", line 126, in run
cpu_dispatch=self.cpu_dispatch, cache_path=opt_cache_path
File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 2451, in new_ccompiler_opt
opt = CCompilerOpt(compiler, **kwargs)
File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 2055, in __init__
_Parse.__init__(self, cpu_baseline, cpu_dispatch)
File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 1624, in __init__
baseline_names = self.feature_names(cpu_baseline)
File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 1182, in feature_names
if self.feature_is_supported(f, force_flags=force_flags):
File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 829, in cache_wrap_me
ccb = cb(self, *args, **kwargs)
File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 1471, in feature_is_supported
if not self.feature_test(impl, force_flags):
File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 829, in cache_wrap_me
ccb = cb(self, *args, **kwargs)
File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 1432, in feature_test
force_flags = self.feature_flags(name)
File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 829, in cache_wrap_me
ccb = cb(self, *args, **kwargs)
File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 1411, in feature_flags
if not f or not self.cc_test_flags(f):
File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 829, in cache_wrap_me
ccb = cb(self, *args, **kwargs)
File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 991, in cc_test_flags
test = self.dist_test(test_path, flags)
File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 556, in dist_test
[source], flags, output_dir=self.conf_tmp_path
File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 536, in dist_compile
sources, extra_postargs=flags, **kwargs
File "C:\Python36\lib\distutils\_msvccompiler.py", line 423, in compile
self.spawn(args)
File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 668, in _dist_test_spawn_paths
self._dist_test_spawn(cmd)
File "D:\Source\numpy\numpy\distutils\ccompiler_opt.py", line 686, in _dist_test_spawn
o = o.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 4: invalid continuation byte
CCompilerOpt._cache_write[786] : write cache to path -> D:\Source\numpy\build\temp.win-amd64-3.6\ccompiler_opt_cache_clib.py
Activity
mattip commentedon Jun 23, 2020
Do you have non-ascii characters in your path? Could you stop at the error and try to work out what is going on?
mattip commentedon Jun 23, 2020
does
ohave an encoding propertyo.encoding?Qiyu8 commentedon Jun 23, 2020
I'm sure that no non-ascii characters existed, It's seems like the exception is triggered when testing SSE flags:
o.decode() failed to decode
eric-wieser commentedon Jun 23, 2020
@Qiyu8, your compiler output is encoded in
GB2312rather than utf8 it seems. The string decodes toeric-wieser commentedon Jun 23, 2020
https://bugs.python.org/issue27179#msg267091 seems slightly related, I haven't read the whole thread.
Qiyu8 commentedon Jun 23, 2020
Maybe a parallel problem, I always build numpy with
python setup.py build_ext -i -j 8, turned topython setup.py build_ext -iand the build successfully.eric-wieser commentedon Jun 23, 2020
My guess would be that the codepage of the parallel processes does not match the codepage of the parent process.
Qiyu8 commentedon Jul 9, 2020
This problem arose again in my local machine, even with non-parallel build.
mattip commentedon Jul 9, 2020
There was a long discussion and a PEP 597 about changing the default stdout encoding. Bottom line: what does
locale.getpreferredencoding()show for you? If you modify the offending lineto use that encoding does it fix the problem?
Qiyu8 commentedon Jul 9, 2020
my encoding result is
cp936, The build can succeed after modifed too.decode(encoding="cp936"), But there are large amount ofwarning D9002 :忽略未知选项“/arch:SSE”produced during build.Qiyu8 commentedon Jul 9, 2020
@mattip I want to provide a PR to solve this issue, modify
decode()todecode(locale.getpreferredencoding()), what do you think?mattip commentedon Jul 9, 2020
I think the preferred spelling of this is when calling
check_outputto specifyuniversal_newlines=Truewhich will force the output to be text (already encoded). Search for "unversal_newlines" on the subprocess doc page. Could you try it (since we don't really test for non-ascii windows versions)?As for the multiple warnings, can you show the part of the log that looks like the one in this comment ?
Qiyu8 commentedon Jul 9, 2020
The build can succeed after specify
universal_newlines=True, see the build log of multiple warnings:mattip commentedon Jul 9, 2020
Good to hear: so the correct fix for this issue is
universal_newlines=TrueAs for the warnings: on 64-bit windows, you should not be using
/arch:SSE2, since it is on by default. This is a separate issue, @seiko2plus might be able to help.eric-wieser commentedon Jul 9, 2020
Note that
universal_newlines=Trueis equivalent to decoding withlocale.getpreferredencoding(False)according to https://docs.python.org/3/library/io.html#io.TextIOWrapper.I still suspect there's a nastier issue here regarding the locale in which parallel vs non-parallel builds are executed in.
seiko2plus commentedon Jul 9, 2020
yes, my bad. I didn't expect msvc to raise warning messages in another language than English.
@Qiyu8, well to solve this issue you have to replace:
numpy/numpy/distutils/ccompiler_opt.py
Line 676 in 86fcce6
with:
And why?
Because microsoft and Intel compiler tending to raise warning messages rather than fatal errros for unkown or unsupported options.