Repository navigation
Drop Solaris support #86339
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2020-11-16.17:05:24.696>
created_at = <Date 2020-10-27.15:59:08.312>
labels = ['interpreter-core', 'invalid']
title = 'Drop Solaris support'
updated_at = <Date 2020-11-16.17:05:24.695>
user = 'https://github.com/vstinner'bugs.python.org fields:
activity = <Date 2020-11-16.17:05:24.695>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-11-16.17:05:24.696>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2020-10-27.15:59:08.312>
creator = 'vstinner'
dependencies = []
files = ['49558', '49559']
hgrepos = []
issue_num = 42173
keywords = ['patch']
message_count = 49.0
messages = ['379767', '379886', '379887', '379888', '379890', '379891', '379893', '379894', '379895', '379897', '379899', '379900', '379901', '379906', '379910', '379915', '379924', '379925', '379926', '379927', '379933', '379937', '379938', '379942', '379946', '379955', '379956', '379959', '379961', '379962', '379963', '379991', '379992', '379993', '379994', '380006', '380008', '380080', '380196', '380203', '380205', '380210', '380241', '380256', '380257', '380455', '380456', '380457', '381122']
nosy_count = 37.0
nosy_names = ['rhettinger', 'terry.reedy', 'jcea', 'vstinner', 'giampaolo.rodola', 'jstasiak', 'pablogsal', 'BTaskaya', 'kulikjak', '[email protected]', 'gaige', 'Yonatan Goldschmidt', 'wiedi', 'mimi.vx', 'jclulow', 'omnios', 'pfirmsto', 'alanc', 'brett3', 'robertdfrench', 'jm650', 'tbalbers', 'mariuspana', 'siepkes', 'cgrzemba', 'marcheschi', 'abarbu', 'predrag.zecevic.1961', 'jorge.schrauwen', 'olafbohlen', 'otis', 'yanchii', 'michael.laspina', 'ijdt.editor', 'jomasoftmarcel', 'jgardner100', 'gwr']
pr_nums = ['23002']
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue42173'
versions = []
Activity
vstinner commentedon Oct 27, 2020
In past years, very few Python core developers took care of the Solaris support. The latest significant Solaris enhancement was in 2018 with the addition of a new Solaris st_fstype attribute to os.stat(), in Python 3.7, by Jesús Cea Avión (bpo-32659).
Another recent Solaris-related change is bpo-41687: "Fix implementation of sendfile to be compatible with Solaris".
On the Oracle side, there was an article about Python on Solaris in 2019:
"Future of Python on Solaris" by Darren Moffat
https://blogs.oracle.com/solaris/future-of-python-on-solaris
I now that Giampaolo Rodola cares of the Solaris support for his psutil project:
https://github.com/giampaolo/psutil/
It's unclear to me if Oracle still plans to maintain Solaris. The latest release was published 2 years ago. The Wikipedia article says: "While Oracle did have a large layoff of Solaris development engineering staff, development continues today of which Solaris 11.4 was released in 2018."
https://en.wikipedia.org/wiki/Solaris_(operating_system)
There is currently 25 open issues which could be closed if we drop the Solaris support.
See also the PEP-11 for the process.
I asked asked 4 years ago (in 2016) if Solaris support should/can be removed:
"OpenIndiana and Solaris support"
https://mail.python.org/archives/list/[email protected]/message/NOT2RORSNX72ZLUHK2UUGBD4GTPNKBUS/
Python has no longer buildbot workers running on Solaris or a Solaris variant (OpenIndiana, Illumos, etc.).
I propose to drop the Solaris support in Python to reduce the maintenance burden.
Python has a few features specific to Solaris:
Solaris has many names and variants:
I'm not sure on how to process? Deprecate all Solaris-specific features in Python 3.10, make sure that Python 3.11 cannot be built on Solaris anymore, and drop the code in Python 3.12? Or drop everything as soon as possible in Python 3.10?
The alternative is to leave the code unchanged and let the code slowly die, as we did for other platforms like HP-UX and IRIX.
See also my notes on platforms supported by Python:
https://pythondev.readthedocs.io/platforms.html
Attached draft PR gives an idea on how much code can be removed if Solaris support is removed. A coase estimation is the removal of 700 lines:
Doc/c-api/init.rst | 3 +-
Doc/distutils/apiref.rst | 1 -
Doc/library/gettext.rst | 10 +-
Doc/library/os.rst | 12 +-
Doc/library/platform.rst | 5 +-
Doc/library/posix.rst | 8 +-
Doc/library/sys.rst | 3 +-
Doc/library/sysconfig.rst | 1 -
Doc/library/time.rst | 13 +-
Doc/using/unix.rst | 7 -
Include/pyport.h | 5 -
Lib/cgi.py | 4 +-
Lib/ctypes/test/test_byteswap.py | 2 +-
Lib/ctypes/util.py | 100 ++--------
Lib/distutils/command/bdist.py | 2 +-
Lib/distutils/tests/support.py | 2 +-
Lib/distutils/tests/test_build_ext.py | 21 --
Lib/distutils/util.py | 11 --
Lib/gettext.py | 3 -
Lib/logging/handlers.py | 2 -
Lib/platform.py | 27 +--
Lib/sysconfig.py | 11 --
Lib/test/fork_wait.py | 4 +-
Lib/test/subprocessdata/fd_status.py | 3 -
Lib/test/test__locale.py | 24 ---
Lib/test/test_asyncio/test_sendfile.py | 6 -
Lib/test/test_asyncore.py | 2 -
Lib/test/test_cmd_line.py | 2 +-
Lib/test/test_curses.py | 2 +-
Lib/test/test_fileio.py | 2 +-
Lib/test/test_importlib/import_/test_path.py | 2 +-
Lib/test/test_locale.py | 2 +-
Lib/test/test_os.py | 14 +-
Lib/test/test_posix.py | 7 +-
Lib/test/test_pty.py | 4 -
Lib/test/test_selectors.py | 2 +-
Lib/test/test_shutil.py | 2 +-
Lib/test/test_socket.py | 4 -
Lib/test/test_strftime.py | 3 +-
Lib/test/test_sys.py | 2 +-
Lib/uuid.py | 5 -
Modules/Setup | 4 -
Modules/_ctypes/ctypes.h | 4 -
Modules/_cursesmodule.c | 2 +-
Modules/_multiprocessing/multiprocessing.h | 15 --
Modules/_posixsubprocess.c | 11 --
Modules/errnomodule.c | 23 +--
Modules/mathmodule.c | 2 +-
Modules/posixmodule.c | 65 +-----
Modules/signalmodule.c | 2 +-
Modules/socketmodule.c | 7 -
Modules/termios.c | 8 +-
Modules/timemodule.c | 9 +-
Python/bootstrap_hash.c | 17 +-
Python/fileutils.c | 3 +-
Python/import.c | 3 +-
Tools/freeze/README | 5 -
Tools/i18n/pygettext.py | 3 +-
Tools/pynche/Main.py | 2 -
Tools/pynche/README | 2 +-
Tools/scripts/nm2def.py | 2 +-
configure | 285 +--------------------------
configure.ac | 101 +---------
pyconfig.h.in | 19 +-
setup.py | 20 +-
65 files changed, 91 insertions(+), 868 deletions(-)
wiedi commentedon Oct 29, 2020
Please don't.
Even if Oracle doesn't care anymore, the open source illumos community does.
We (illumos) share the same uname "SunOS" and depend on this support.
I have offered to host an illumos buildbot in the past and that offer is still valid.
At that time the testsuite did not pass and I could not figure out the fix yet.
If having this fixed is a requirement to keep illumos supported I will invest more time into this. I can also offer shell access to anyone interested in helping with this.
vstinner commentedon Oct 29, 2020
I looked at PyPI download statistics of the "six" module over the last 180 days (6 months), grouped by operating system name:
SunOS is 0.0007% of all six downloads.
For comparison, the top 4 is:
Raw data of the top 14:
num_downloads,system_name
535804817,Linux
21053173,Windows
18695360,
14776993,Darwin
347090,FreeBSD
9759,Qnx
5350,Java
3975,SunOS
3178,CYGWIN_NT-10.0-14393
3177,CYGWIN_NT-10.0-17763
2505,CYGWIN_NT-10.0-18363
2263,AIX
2061,NetBSD
1891,OpenBSD
vstinner commentedon Oct 29, 2020
Which Python version do you use?
https://blogs.oracle.com/solaris/future-of-python-on-solaris mentions Python 2.7 and 3.4.
wiedi commentedon Oct 29, 2020
Myself currently mostly python 3.8 via pkgsrc on SmartOS (one of the illumos distributions).
OmniOS ships 3.7: https://github.com/omniosorg/omnios-build/tree/master/build/python37
OpenIndiana ships at least 2.7, 3.5, 3.7.
The illumos build process also depends on python and was updated to support 3.9 two weeks ago: https://www.illumos.org/issues/13217
mimivx commentedon Oct 29, 2020
Oracle moved solaris 11.4 to +- rolling release mode, last SRU is 26 from 20.10.2020 with month release cadence.
On another side, almost all informations are behind the Oracle paywall
jcea commentedon Oct 29, 2020
I think Solaris Support is still important and I could invest more time on it.
About package downloads:
I routinely download packages in my laptop (Linux) and distribute them to a fleet of servers (Linux and Solaris) via ansible and similar. Many servers in intranets don't install packages directly via PIP.
Under OmniOS or SmartOS (Illumos distributions), many people use the system package manager to install Python packages, not "PIP". For instance, http://netbsd.2816.n7.nabble.com/WIP-commit-Upgrade-to-Mailman-2-1-34-td427426.html.
Buildbots: I used to manage a buildbot with OpenIndiana, but it didn't survived the GITHUB migration. I could provide now a SmartOS buildbot.
As a core developer, I do care about Solaris. I fully realize I have been quite inactive since the github migration. My fault. I could devote more time to the task.
jcea commentedon Oct 29, 2020
I am using Python 3.8 installed via pkgsrc in SmartOS with some Zones running Python 3.9 self compiled code.
More about the downloads stats:
4% packages in windows: I am pretty sure that most windows programmers install packages via conda-like tools. Surely there is more that 4% of python programmers on windows, even more in machine learning and data science areas, where windows and MacOS X rule (apparently being only 7% of PIP downloads).
Even if the PIP download stats would reflect penetration, Solaris is ahead of NetBSD and OpenBSD together.
jclulow commentedon Oct 29, 2020
Hi! I'm a member of the illumos core team, and I'm also pretty keen for us to keep Python support! Some of our core OS build and packaging tooling is written in Python, and certainly applications like Synapse (Matrix) and Review Board are important to myself.
We currently provide a virtual machine for builds to the Go project that is wired in to their official CI/CD. I expect we can help out with a similar thing here!
I know some illumos distributions (OmniOS, OpenIndiana, anything with pkgsrc, etc) are maintaining some patches against the stock Python to fix failing tests, etc. Hopefully we can figure out how to get those things upstreamed and increase our participation there.
citrus-it commentedon Oct 29, 2020
We at OmniOS (an illumos distribution) are in the process of upgrading to Python 3.9 and working on getting tests clean. We're in pretty good shape with a few local patches and only having to skip a few tests at the moment, and most of those patches should be suitable for integrating generally.
Some of the tests just need adjusting to be aware of the different platform (e.g. illumos does not support multiple SCM_RIGHTS messages in a single packet), some still need investigating to find the root cause, but the list is small.
citrus-it commentedon Oct 29, 2020
Re: Downloads
OmniOS makes modules like 'six' available through its own IPS packaging system (which is written in Python) so installations won't show up pypi stats.
six is a core module on the platform so it is installed on 1000s of OmniOS machines (and then large numbers of SmartOS, OpenIndiana and tribblix ones too - all using the SunOS tag). The numbers aren't going to be in the same league as other platforms but the downloads stats definitely don't represent the illumos installation base and dependency on Python.
30 remaining items
jgardner100 commentedon Nov 2, 2020
Don't remove, it is still actively used by so many people. If the maintance overhead of 700 lines is a large burdern, then I'm happy to take on whatever work is required for it.
citrus-it commentedon Nov 2, 2020
Jakub's results looks very familiar to me, having been working on python 3.9 on illumos.
For OmniOS, we currently skip these tests via --ignorefile:
# wchar_t related failures
*.test_re.ReTests.test_locale_compiled
*.test_re.ReTests.test_locale_caching
# illumos does not support multiple SCM_RIGHTS messages in a packet
*FDPassSeparate*
# These tests fail on illumos and the first consumes a significant
# amount of memory. Investigation required.
test.test_socket.SendfileUsingSendfileTest.testCount
test.test_socket.SendfileUsingSendfileTest.testWithTimeout
test.test_socket.SendfileUsingSendfileTest.testOffset
test.test_asyncio.test_sendfile.*
test.test_asyncio.test_subprocess.SubprocessMultiLoopWatcherTests.*
We are also carrying some local patches for the following. Several come from Solaris.
Stop python detecting and using epoll() on illumos
scheduling priorities can be < 0
https://github.com/citrus-it/omnios-build/blob/python39/build/python39/patches/mod-posix-sched_priority.patch
differences in sendfile behaviour (improves the situation but is not complete)
https://github.com/citrus-it/omnios-build/blob/python39/build/python39/patches/mod-posix-sendfile.patch
Enable sendfile for shutil.copy (mismatch between library and testsuite in terms of whether sendfile() is enabled on illumos)
https://github.com/citrus-it/omnios-build/blob/python39/build/python39/patches/mod-shutil-sendfile.patch
Fixes for building the socket module (_XOPEN_SOURCE=600)
https://github.com/citrus-it/omnios-build/blob/python39/build/python39/patches/mod-socket-xpg6.patch
Emulate clock_gettime(CLOCK_THREAD_CPUTIME_ID)
https://github.com/citrus-it/omnios-build/blob/python39/build/python39/patches/mod-time-threadtime.patch
PTY patch
https://github.com/citrus-it/omnios-build/blob/python39/build/python39/patches/pty.patch
and a few others, not all of which are suitable for upstream.
https://github.com/citrus-it/omnios-build/tree/python39/build/python39/patches
With all of this in place, the headline test stats for OmniOS Python 3.9 are:
401 tests OK.
24 tests skipped:
test_dbm_gnu test_epoll test_gdb test_idle test_kqueue test_msilib
test_ossaudiodev test_smtpnet test_socketserver test_startfile
test_tcl test_timeout test_tix test_tk test_ttk_guionly
test_ttk_textonly test_turtle test_urllib2net test_urllibnet
test_winconsoleio test_winreg test_winsound test_xmlrpc_net
test_zipfile64
Tests result: SUCCESS
and, additionally, the dtrace tests succeed (we test them separately as they require elevated privileges).
kulikjak commentedon Nov 2, 2020
Thanks Andy, good to know we are seeing the same issue!
We tried to fix sendfile differences in Python code before as well, but as you said, it was never 100% solved. Recently we finally fixed it in C and it was accepted (https://bugs.python.org/issue41687) and backported to 3.9 so you might/should be ok without the patch as well.
CLOCK_THREAD_CPUTIME_ID was issue for us as well but before the patch was accepted upstream, Oracle Solaris implemented it so it was no longer necessary. But the issue is still open https://bugs.python.org/issue35455 and knowing that others will use it, I will dust it off and finish it.
Knowing that Oracle and Illumos are facing the same issues, I am much more confident that our patches won't break Illumos. I will start upstream more of them right away.
gwr commentedon Nov 2, 2020
I can understand the frustrations around dealing with minority platforms, but please remember that the illumos project (www.illumos.org) is basically the inheritor of problems around "Build on Solaris" for 3rd party software like Python. There are several OS distributions based on illumos that would be impacted by removing the ability to (easily) build Python. I'm confident we can find some people to help maintain build-bots etc. if that's what it takes. If you're still looking for such help, let me know and I'll ask on [email protected] for volunteers.
Thanks! -GWR
terryjreedy commentedon Nov 2, 2020
People interested in helping Solaris issues should see my post above, msg380006.
Issue bpo-35455, about thread time on Solaris, with a simple but non-trivial patch, was just closed as fixed after reviews from 2 coredevs and a 3rd person.
I presume Victor will close his proposed PR when there is a buildbot and additional action on issues. We have gotten new information that supports doing so.
rhettinger commentedon Nov 2, 2020
Victor, are you going to close the PR?
vstinner commentedon Nov 6, 2020
Copy of my email sent to python-dev.
https://mail.python.org/archives/list/[email protected]/message/46UUJ4J5YLCWF2JQMC5L7OIYG6PNQLRL/
Since I created the issue and the PR, and sent this email to
python-dev (one week ago), many Solaris and Solaris-like (ex: Illumos)
users replied that the operating system is definitively alive. I
didn't know that Oracle still ships new Solaris updates every month:
that's a good thing!
But this is not enough to support a platform. We would need proactive
contributors to fix known Solaris issues, but also fix new Solaris
issues (either regressions, or bugs newly discovered). We would also
need a buildbot to run the Python test suite on Solaris (or again, a
Solaris-like OS).
The good news is that Jakub Kulik started to fix some Solaris issues.
I understood that Solaris and Solaris-like operating systems do have
downstream patches on Python to fix a bunch of bugs. It seems like
some people want to push these fixes to Python upstream which is also
a good sign.
The other problem that I wanted to discuss is that fixing Solaris
issues require core devs (who merge PRs) accessing Solaris. If
contributors send patches and some core devs are fine with merging
fixes without being able to test them manually, I'm also fine with
that.
My first intent was to remove support for a definitely dead operating
system, but it seems like I was completely wrong (it's alive!). Thanks
to people starting to fix Solaris issues, I close my PR and I no
longer plan to drop Solaris support. I prefer to leave bpo-42173 open
for now, since people decide to use it as a place to collaborate on
fixing Solaris issues.
Once most tests will pass on the master branch, I also hope that
someone will set up a buildbot *and* fix issues discovered by this
buildbot. Sorry but just setting up a buildbot doesn't solve any
problem, it only increases the maintenance burden for people who
maintain the buildbot fleet. For example, we have two AIX buildbots, I
report bugs on bugs.python.org, but it seems like nobody is available
to fix them...
Overall, I'm quite happy with what is happening with Solaris! More
collaboration, issues being fixed in Python upstream. I just hope that
this work will continue next months. ;-)
vstinner commentedon Nov 6, 2020
This issue title is "Drop Solaris support". I no longer plan to drop Solaris support. I suggest to open new issues and continue the discussion there.
vstinner commentedon Nov 6, 2020
Gaige B Paulsen offered to set up a SmartOS (Illumos-derived) buildbot:
https://mail.python.org/archives/list/[email protected]/thread/7HGPHD3WZG6V6NVP6EGQO6NHZ3DAPPXN/
vstinner commentedon Nov 16, 2020
I close the issue. Thanks for the constructive discussion and I hope that more and more downstream patches will be merged in Python upstream ;-)