Repository navigation
Update PCBuild for VS 2015 #67108
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/zooba'
closed_at = <Date 2015-01-17.23:11:30.791>
created_at = <Date 2014-11-22.21:40:07.139>
labels = ['type-feature', 'OS-windows', 'build']
title = 'Update PCBuild for VS 2015'
updated_at = <Date 2015-11-06.18:08:00.514>
user = 'https://github.com/zooba'bugs.python.org fields:
activity = <Date 2015-11-06.18:08:00.514>
actor = 'steve.dower'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2015-01-17.23:11:30.791>
closer = 'steve.dower'
components = ['Build', 'Windows']
creation = <Date 2014-11-22.21:40:07.139>
creator = 'steve.dower'
dependencies = []
files = ['37246', '37247', '37248', '37251', '37265', '37270', '37374', '37392', '37405']
hgrepos = []
issue_num = 22919
keywords = ['patch', 'needs review']
message_count = 31.0
messages = ['231531', '231533', '231541', '231547', '231593', '231604', '231631', '231634', '231635', '231878', '232237', '232239', '232248', '232249', '232344', '232410', '232570', '232571', '233637', '233641', '233643', '233644', '233648', '233652', '233671', '233945', '233946', '233969', '234202', '254205', '254206']
nosy_count = 11.0
nosy_names = ['pitrou', 'vstinner', 'tim.golden', 'jkloth', 'trent', 'BreamoreBoy', 'python-dev', 'zach.ware', 'mdengler', 'steve.dower', 'Kevin.Phillips']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue22919'
versions = ['Python 3.5']
Activity
zooba commentedon Nov 22, 2014
I've basically finished the core of the work to refresh the PCBuild project files and support building with VS 2015, and I believe it's ready to merge in.
Though the title says VS 2015, builds will work fine with VS 2010 and VS 2013 (and probably VS 2012, but I didn't try it). Even so, I've copied the old project files into PC/VS10.0, though I'd be happy to just forget them.
I suspect Tools/msi/msi.py will stop working with this change. My replacement installer is not ready yet. If anyone's particularly concerned about msi.py then I can try and restore it before checking this in, but I'm 99.9% certain it won't be used for 3.5, so I don't see the point.
I've attached two patches, one with all the changes and the other with the "highlights" - the diff between the diffs are most of the project files and the added/deleted files. I imagine practically everyone will have a better time viewing the changes in their own hg tools, so the changes can be pulled from my sandbox with "hg pull https://hg.python.org/sandbox/steve.dower -b Projects" or viewed at https://hg.python.org/sandbox/steve.dower/shortlog/d08b456124e5. (I'll rebase these as a single commit in default when it goes in.)
A few changes that may deserve some more discussion:
Builds for 32-bit now go to PCBuild\win32 instead of PCBuild (the batch file updates should make this transparent)
The project that used to trigger the OpenSSL build is now two projects (libeay, ssleay) that do the build directly, so they may need updating whenever we update the OpenSSL version or change _ssl/_hashlib. I'm obviously happy to track these. (The advantages are proper incremental builds, better debugging, better optimisation - definitely PGO if we turn that back on.)
The posixmodule.c update for Py_Verify_fd may become redundant - the VC14 CRT will include a per-thread function to change the invalid parameter handling, so when that's available we should be able to switch away from this trickery completely.
I changed some GetVersion calls (which no longer behave properly as of Windows 8.1) to use functions from VersionHelpers.h in VS 2013 and later.
build.bat now builds in parallel by default, with '-M' to disable.
PCBuild/readme.txt has been updated as if VS 2015 has already been released. This is not yet true, but I doubt anyone who notices will be particularly confused or upset.
zooba commentedon Nov 22, 2014
Oh, one other thing I just thought of: the current release of nmake (in VS 2015) has a regression that will make TCL and TK fail to build, but I have a workaround for their makefile. OpenSSL also had a problem with VC14 but their fix isn't in the build we've got on svn.python.org.
If possible, I'd like to get the workarounds into our sources for those before this is checked in, so that the buildbots (currently there's one that I know of with VS 2015 Preview installed) can pick up those fixes and won't fail because of these.
The nmake issue should be fixed for VS 2015 RC and the OpenSSL issue is already fixed (in their 1.1 branch, IIRC). I've attached a patch listing the changes needed.
zooba commentedon Nov 23, 2014
Fixed and pushed a minor issue with debug builds - I wasn't setting the 'g' suffix on the tcltk libs correctly.
zooba commentedon Nov 23, 2014
Pushed some changes based on Zach's feedback and added a new complete patch (minus the deletion of the PC/VS9.0 and PC/VS10.0 folders).
zooba commentedon Nov 24, 2014
Added some fixes to the python3.dll build (and xxlimited test project) which I noticed thanks to Antoine's feedback.
zware commentedon Nov 24, 2014
Another round of review posted on Rietveld. I'll try to get the VS2015 preview downloaded today and give some more feedback after using it.
Just one inconsequential note on python3.diff: the 'lib' command in python3dll.vcxproj could use a '/nologo' flag.
zware commentedon Nov 25, 2014
After a debug build with VS2015, my biggest concern is this:
..\PC\bdist_wininst\install.rc(19): fatal error RC1015: cannot open include file 'afxres.h'. [P:\ath\to\cpython\PCbuild\bdist_wininst.vcxproj]
There's also a slew of new warnings, mostly C4456 and C4457 with a few C4459s thrown in. I think we can safely disable those three (which are all about name shadowing), perhaps with a XXX comment alongside to the effect of 'it might be nice to not disable these warnings someday'.
Interestingly, there's also this warning:
C:\Program Files (x86)\Windows Kits\8.1\Include\um\shlobj.h(1054): warning C4091: 'typedef ': ignored on left of 'tagGPFIDL_FLAGS' when no variable is declared [P:\ath\to\cpython\PCbuild\pylauncher.vcxproj]
And a matching one for pywlauncher.vcxproj.
zooba commentedon Nov 25, 2014
I thought I had a workaround for this, but I'll have to check it out. Oddly enough, all of my builds have been fine (I think I'm mostly doing release builds - does that make a difference?)
I'm torn about that. Ideally I'd like to suppress the existing ones and still have new ones appear, but that probably requires going through and fixing them. Maybe leaving the warnings there will encourage someone to do it? :)
This is a known issue with the Windows SDK - they need to release a new version to handle the changes in the compiler. It doesn't affect the Python build, though it is currently breaking some of my installer work :( I'm confident it will be fixed soon - they're getting a lot of reports about it.
zooba commentedon Nov 25, 2014
Added changes from Zach's last review.
Highlight: I deleted the make_versioninfo project and build step :)
zooba commentedon Nov 30, 2014
I haven't been able to reproduce this. bdist_wininst isn't selected to build for debug in VS, but it builds fine for me anyway. Was your repo dirty? (It didn't make a difference for me, but I have had files occasionally not update properly when switching branches.)
zooba commentedon Dec 6, 2014
Does anyone else want to provide any feedback? Or should I go ahead and get it in (after rebasing/retesting/etc.)? The feedback so far has been really helpful.
9 remaining items
vstinner commentedon Jan 8, 2015
Oh by the way, 2 Windows buildbot slaves are no more able to compile Python. I don't know if it's related to this issue or not.
Builder "AMD64 Windows7 SP1 3.x", owned by Jeremy Kloth:
http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x
2>LINK : fatal error LNK1104: cannot open file 'C:\buildbot.python.org\3.x.kloth-win64\build\PCBuild\amd64\python35_d.dll' [C:\buildbot.python.org\3.x.kloth-win64\build\PCbuild\pythoncore.vcxproj]
Builder "x86 Windows Server 2003 [SB] 3.x" managed by Snakebite.org (trent):
http://buildbot.python.org/all/builders/x86%20Windows%20Server%202003%20%5BSB%5D%203.x
E:\Data\buildslave\cpython\3.x.snakebite-win2k3r2sp2-x86\build\Tools\buildbot\..\..\build\TEEB4C~1 - The process cannot access the file because it is being used by another process.
(...)
2>E:\Data\buildslave\cpython\3.x.snakebite-win2k3r2sp2-x86\build\PCbuild\pythoncore.vcxproj(415,5): warning : Toolset v100 is not used for official builds. Your build may have errors or incompatibilities.
(...)
C:\Program Files\MSBuild\Microsoft.Cpp\v4.0\Platforms\Win32\Microsoft.Cpp.Win32.Targets(147,5): error MSB6006: "CL.exe" exited with code 128. [E:\Data\buildslave\cpython\3.x.snakebite-win2k3r2sp2-x86\build\PCbuild\pythoncore.vcxproj]
Both slaves are part of the group of "3.x stable" buildbots :-(
pitrou commentedon Jan 8, 2015
Visual Studio 2013 Professional works fine under Windows 7 SP1 here.
vstinner commentedon Jan 8, 2015
Ok, good to know. But is it correct that the free version of VS 2013 (community) requires Windows 8.1 or newer? It's not cool to require to upgrade Windows to being able to freely compile Python.
My MSDN account is probably outdated, I don't think that I have access to the Professional version.
Would it be possible to build Python with the light Windows SDK which basically only includes the C compiler and the CRT?
BreamoreBoy commentedon Jan 8, 2015
@victor I don't know what version you need for Windows 7 or earlier but I can tell you that VS 2013 Community edition is *NOT* free, I fell into that trap myself, you need the Express edition.
zooba commentedon Jan 8, 2015
You will need Windows 7 *SP1*, as I don't think VS will run without the updates. There is also a service pack for VS 2010 that may enable opening the newer solution - it certainly worked for me.
We decided not to keep the old project files as they weren't being maintained.
Community edition is free, but you need a Microsoft account (also free) to register it. I don't know why this is necessary, but that's the way they set it up.
I'll check out that buildbot when I get a chance (Trent's one is known to be busted). The stable buildbots and the one with VS 2015 were doing fine last time I looked.
I haven't tried with the SDK compilers, but it should be possible to build with them now. VS is no longer required to build from the command line.
zooba commentedon Jan 8, 2015
Just emailed Jeremy about the buildbot. It looks like the last time tests were run something didn't clean up properly and left the build output locked. There's nothing wrong with the project files.
vstinner commentedon Jan 13, 2015
I do have Windows 7 SP1.
Oh, I also worked for me. I have VS 2010 *Express* and I'm now able to open the project after installing the Service Pack 1 of VS 2010.
Note: The binary (in debug mode) moved from PCbuild\python_d.exe to PCbuild\win32\python_d.exe
zooba commentedon Jan 13, 2015
I expected so, but didn't want to assume.
Glad to hear it. You made me a little nervous there :) (I don't feel like requiring VS 2010 SP1 is unreasonable, but I could be wrong... not sure there's any sensible way to support VS 2010 RTM though.)
Yes, that was always a pet peeve of mine, but I got big thanks from the other Windows nosy list for making that move. At this point, everything should be okay with the new location, but if there are any more issues just let me know and I'll check them out.
zware commentedon Jan 13, 2015
STINNER Victor added the comment:
There ought to be a 'python.bat' in the root of the source tree that
will always point to the last-built python[_d].exe, which may ease
things for you.
zooba commentedon Jan 17, 2015
Closing again, as Victor's issue was resolved (VS 2010 SP1 is needed, and I'm updating the devguide to specify that on bpo-23257).
KevinPhillips commentedon Nov 6, 2015
This change has the adverse side effect of requiring users of Python 3.5 to use the Visual Studio 2015 runtime library. Further, as is documented on Microsoft's website, the minimum system requirements for installing and running this runtime is Windows 7 SP1 or newer.
As a result, adopting this new compiler for the Python runtime prevents users of operating systems older than this from using this version. Even users with Windows 7 pre service pack are unable to use it. Based on a quick review of the comment thread here, the choice for adopting the 2015 compiler was relatively arbitrary and thus an older version, say 2013 for example, could have been used instead and older operation systems would still be supported with little to no impact on Python.
While this is unfortunate to say the least, what makes matters worse is there appears to be little to no information on the Python.org website indicating that the minimum system requirements have now been affected in this way. Also, for users attempting to install v3.5 on a system older than this are presented with a cryptic error message basically just saying that "some unknown error has occurred", leaving them scratching their head trying to figure out what's wrong.
Seeing as how you can't go back in time and fix this in 3.5.0, I am hoping that at the very least you could add some information about this to some obvious location on the main Python.org website, and maybe consider updating the installation for future 3.5.x releases such that a check is done to see if the operating system meets these new minimum requirements and present the user with some helpful information that would lead them to a resolution.
zooba commentedon Nov 6, 2015
The check is already added and will be in 3.5.1.
Changes to the python.org website should be suggested on https://github.com/python/pythondotorg (but I agree it could be more clear on the main site - the rule that we only support the same versions of Windows that Microsoft supports is not very well known, unfortunately).