Removing Win386 executables from images #503

Activity

1. tianon commented on Jul 6, 2020

   

   tianon

   on Jul 6, 2020

   Member

   More actions

   I'm honestly a bit on the fence with this one, but here's what makes me think it might be appropriate for us to leave them as-is:

   1. they're part of the "standard distribution" of PIP

   2. they're relatively small (~2.8M all together)

   I have to believe there's been some thought in their relevant upstreams as to whether or not to have pip and/or python distributions not include them for non-Windows platforms. 🤔
2. hakandilek commented on Jul 8, 2020

   

   hakandilek

   on Jul 8, 2020

   Author

   More actions

   1. (Speaking about distutils/wininst) I don't feel comfortable from the OSS Licensing (and with the security as well) perspective, distributing a binary which I haven't compiled myself or not sure if the binaries match with the sources. Considering that some of those files date back to 2008-2009, you might understand my concern here (no offense meant to core python developers committing those files).

   2. Here comes my nerdy 🤓 argument:

      2.8MB seems to be small but if you consider an image like docker which is downloaded over 500M+ times (wish I had the weekly/monthly numbers) this makes around 1.4PB of bandwidth. This is just the downstream traffic from Docker Hub. Considering the huge amount of storage space and bw scattered around the globe we can save a lot with just removing couple of MBs. 🦄
3. tianon commented on Jul 8, 2020

   

   tianon

   on Jul 8, 2020

   Member

   More actions

   I did some digging upstream and found the following useful links:

   • https://bugs.python.org/issue37468
   • bpo-37468: make install no longer install wininst-*.exe files python/cpython#14511

   So, in Python 3.9+, the wininst-*.exe files are removed from Linux installations (and both Debian and Fedora remove them as well, lending more credence to doing so, although I find it very amusing that RHEL does not with roughly the same justification I've given above 😅).

   Looking at the setuptools binaries (gui*.exe and cli*.exe), those are included in the relevant Debian package (https://packages.debian.org/sid/all/python3-setuptools/filelist), so I do not doubt that they are probably meaningful even on Linux (and thus we should not touch those). I was not able to find an issue on setuptools discussing it (although that earlier thread references them and states that they're out of scope for that particular issue, which is fair).

   Digging in distlib, Debian does remove those (https://sources.debian.org/src/distlib/0.3.0-1/debian/README.Debian/, https://sources.debian.org/src/distlib/0.3.0-1/debian/patches/no-binaries.diff/), but I wasn't able to find any detailed rationale nor any upstream issue discussing them in any way. 😞
4. hakandilek commented on Jul 13, 2020

   

   hakandilek

   on Jul 13, 2020

   Author

   More actions

   Thanks for the research. So, what should we do?

   My suggestion would be:

   • wininst: remove from all supported images (3.6+ including debian and alpine) considering there's no real reason to keep them.

   • setuptools: leaving them in the image. They might be still in use for cross-compilation purposes or on Cygwin or MinGW.

   • distlib: remove from Debian based images (3.6+)
5. hakandilek commented on Jul 24, 2020

   

   hakandilek

   on Jul 24, 2020

   Author

   More actions

   Any other ideas about this?
   Should I update #504 as I suggested before?
6. tianon commented on Jul 31, 2020

   

   tianon

   on Jul 31, 2020

   Member

   More actions

   Sorry for the delay -- I think we've got strong upstream-supported evidence that wininst-*.exe are safe/valid for us to remove (and that we don't need to do so in Python 3.9+).

   I do not think the evidence for distlib is quite as strong -- the fact that Debian removes them is a compelling data point, but doesn't give us any clues about what upstream supports/prefers, so I'd rather we leave them alone until/unless we can get a more conclusive upstream opinion on whether they're officially safe for us to remove.

   I was in this code making a fix for #507, so I've included the wininst-*.exe deletion code for Python 3.8 and lower in #508. 😅

   Thanks for the contribution, and for starting/driving the conversation!
7. yosifkit

   closed this as completedin #508on Aug 3, 2020