Skip to content

Potentially merge Python 2 and 3 stubs for many modules #4

Closed
Closed
Potentially merge Python 2 and 3 stubs for many modules#4
Labels
stubs: improvementImprove/refactor existing annotations, other stubs issues

Description

@JukkaL
JukkaL
opened on Apr 13, 2015

We'd need to add aliases for string-like types such as str/bytes/unicode, and these would be different between Python 2 and 3. We'd need to decide where to define the type aliases, either in the stub of the module that uses them or in a shared typeshed-related (?) module that is only available as a stub.

Activity

  1. vlasovskikh commented on Apr 13, 2015

    @vlasovskikh
    vlasovskikh
    on Apr 13, 2015
    Member

    Here is what we found useful in PyCharm (written using our old syntax):

    string             # Py2: str | unicode, Py3: str
bytestring         # Py2: str | unicode, Py3: bytes
bytes              # Py2: str, Py3: bytes
unicode            # Py2: unicode, Py3: str

  2. matthiaskramm commented on Apr 17, 2015

    @matthiaskramm
    matthiaskramm
    on Apr 17, 2015
    Contributor

    Sharing code between pyi files sounds even more useful for https://github.com/JukkaL/typeshed/issues/3
    Imagine a package with a lot of files (like e.g. numpy), and only two or three change between versions.
    In terms of storing a duplicate modules in the github repo, we could just use symlinks.

    However, concerning type aliases, I'm not sure how often we'll see the case that the Python 2 and the Python 3 version off the stub are exactly the same, except one uses unicode and the other uses str.

  3. gvanrossum commented on Jan 26, 2016

    @gvanrossum
    gvanrossum
    on Jan 26, 2016
    Member

    Once mypy (and other systems) support version checks as described in PEP 484 (https://www.python.org/dev/peps/pep-0484/#version-and-platform-checking) we should be able to unify many more stubs.

  4. gvanrossum
    added
    help wanted
    on Nov 4, 2016
  5. gvanrossum
    removed
    help wanted
    on Apr 30, 2017
  6. cocoatomo
    added a commit that references this issue on Jul 17, 2018

    Merge pull request python#4 from cocoatomo/update-master

    27530aa
  7. srittau
    added
    size-large
    on Sep 11, 2018

  8. srittau commented on Sep 11, 2018

    @srittau
    srittau
    on Sep 11, 2018
    Collaborator

    Here is a list of modules that a simple diff between stdlib/2 and stdlib/3 uncovered (please edit as appropriate):

    • _ast.pyi
    • _threading_local.pyi
    • _warnings.pyi
    • abc.pyi
    • ast.pyi
    • atexit.pyi
    • builtins.pyi
    • compileall.pyi
    • email
    • encodings
    • fcntl.pyi
    • fnmatch.pyi
    • functools.pyi
    • gc.pyi
    • getopt.pyi
    • getpass.pyi
    • gzip.pyi
    • hashlib.pyi
    • heapq.pyi
    • imp.pyi
    • inspect.pyi
    • io.pyi
    • itertools.pyi
    • _json.pyi
    • macpath.pyi
    • multiprocessing
    • ntpath.pyi
    • nturl2path.pyi
    • os (see Merge os stubs into 2and3 #1427)
    • pipes.pyi
    • platform.pyi
    • posixpath.pyi
    • posix.pyi (see Fill in stdlib/3/posix.pyi #616)
    • random.pyi
    • re.pyi
    • resource.pyi
    • runpy.pyi
    • shelve.pyi
    • shlex.pyi
    • signal.pyi
    • smtplib.pyi
    • spwd.pyi
    • sre_constants.pyi
    • sre_parse.pyi
    • stat.pyi
    • string.pyi
    • subprocess.pyi
    • symbol.pyi
    • sys.pyi
    • tempfile.pyi
    • textwrap.pyi
    • tokenize.pyi
    • types.pyi
    • typing.pyi

    The packages (marked in italics) need special consideration, since they might contain submodules that don't exist in both versions.

  9. srittau commented on Oct 23, 2018

    @srittau
    srittau
    on Oct 23, 2018
    Collaborator

    Also json, where the stub is currently a module in stdlib/2, but a package in stdlib/3. In the standard library it is a package in both.

  10. srittau
    added
    stubs: improvementImprove/refactor existing annotations, other stubs issues
    on Oct 28, 2018

  11. hauntsaninja commented on Jun 11, 2020

    @hauntsaninja
    hauntsaninja
    on Jun 11, 2020
    Collaborator

    Can we close this?

    In 2020, it seems like it could be unnecessary burden (and potentially cause errors in py2 stubs by py3-minded people) to merge py2 and py3 stubs using version checks. It's a different matter if the files are identical, but a quick command makes it seem like that's not really the case, see details.

    Details

    Files which have the same content hash, compare to what's enforced by check_consistent.py

    ~/dev/typeshed master λ fd . --type f | xargs md5 -s | pyp -b 'd = defaultdict(list)' 'z = x.split(); d[z[-1]].append(z[1])' -a '[v for v in d.values() if len(v) > 1]'
['(stdlib/2/_winreg.pyi)', '(stdlib/3/winreg.pyi)']
['(stdlib/2/copy_reg.pyi)', '(stdlib/3/copyreg.pyi)']
['(stdlib/2/os2emxpath.pyi)', '(stdlib/3/os/path.pyi)', '(stdlib/2/os/path.pyi)', '(stdlib/2and3/ntpath.pyi)', '(stdlib/2and3/posixpath.pyi)']
['(stdlib/2/__builtin__.pyi)', '(stdlib/2and3/builtins.pyi)']
['(stdlib/3/enum.pyi)', '(third_party/2/enum.pyi)']
['(stdlib/3/xmlrpc/__init__.pyi)', '(stdlib/3/email/mime/__init__.pyi)', '(stdlib/3/concurrent/__init__.pyi)', '(stdlib/2/email/mime/__init__.pyi)', '(stdlib/3/urllib/__init__.pyi)', '(stdlib/2/distutils/__init__.pyi)', '(third_party/2and3/dateutil/__init__.pyi)', '(third_party/2and3/geoip2/__init__.pyi)', '(third_party/2and3/tornado/__init__.pyi)', '(stdlib/2and3/wsgiref/__init__.pyi)', '(third_party/2and3/pyVmomi/__init__.pyi)', '(stdlib/2and3/xml/etree/__init__.pyi)', '(third_party/2and3/requests/packages/urllib3/packages/__init__.pyi)', '(third_party/2and3/requests/packages/urllib3/contrib/__init__.pyi)', '(stdlib/2and3/distutils/__init__.pyi)', '(third_party/2and3/werkzeug/contrib/__init__.pyi)', '(stdlib/2and3/distutils/command/bdist.pyi)', '(third_party/2and3/werkzeug/middleware/__init__.pyi)', '(stdlib/2and3/distutils/command/install_lib.pyi)', '(stdlib/2and3/distutils/command/clean.pyi)', '(stdlib/2and3/distutils/command/bdist_rpm.pyi)', '(stdlib/2and3/distutils/command/install_data.pyi)', '(stdlib/2and3/distutils/command/build_ext.pyi)', '(stdlib/2and3/distutils/command/check.pyi)', '(stdlib/2and3/distutils/command/build_scripts.pyi)', '(stdlib/2and3/distutils/command/install_scripts.pyi)', '(stdlib/2and3/distutils/command/install_headers.pyi)', '(stdlib/2and3/distutils/command/build.pyi)', '(stdlib/2and3/distutils/command/bdist_packager.pyi)', '(stdlib/2and3/distutils/command/__init__.pyi)', '(stdlib/2and3/distutils/command/bdist_dumb.pyi)', '(stdlib/2and3/distutils/command/register.pyi)', '(stdlib/2and3/distutils/command/bdist_wininst.pyi)', '(stdlib/2and3/distutils/command/build_clib.pyi)', '(stdlib/2and3/distutils/command/sdist.pyi)', '(third_party/2and3/cryptography/__init__.pyi)', '(third_party/2and3/google/protobuf/util/__init__.pyi)', '(third_party/2and3/cryptography/hazmat/bindings/openssl/__init__.pyi)', '(third_party/2and3/cryptography/hazmat/bindings/__init__.pyi)', '(third_party/2and3/google/protobuf/compiler/__init__.pyi)', '(third_party/2and3/google/protobuf/internal/__init__.pyi)', '(third_party/2and3/google/__init__.pyi)', '(third_party/2and3/pymysql/constants/__init__.pyi)', '(third_party/2/fb303/__init__.pyi)', '(third_party/2/kazoo/__init__.pyi)', '(third_party/2/kazoo/recipe/__init__.pyi)', '(third_party/2/OpenSSL/__init__.pyi)', '(third_party/2/scribe/__init__.pyi)', '(third_party/2/concurrent/__init__.pyi)', '(third_party/2and3/Crypto/Random/Fortuna/__init__.pyi)', '(tests/mypy_blacklist.txt)', '(third_party/3/jwt/contrib/__init__.pyi)']
['(stdlib/2/email/MIMEText.pyi)', '(stdlib/2/email/mime/text.pyi)']
['(stdlib/3/concurrent/futures/__init__.pyi)', '(third_party/2/concurrent/futures/__init__.pyi)']
['(stdlib/3/concurrent/futures/process.pyi)', '(third_party/2/concurrent/futures/process.pyi)']
['(stdlib/3/concurrent/futures/_base.pyi)', '(third_party/2/concurrent/futures/_base.pyi)']
['(stdlib/3/concurrent/futures/thread.pyi)', '(third_party/2/concurrent/futures/thread.pyi)']
['(stdlib/3/ipaddress.pyi)', '(third_party/2/ipaddress.pyi)']
['(stdlib/3/unittest/mock.pyi)', '(third_party/2and3/mock.pyi)']
['(stdlib/3/pathlib.pyi)', '(third_party/2/pathlib2.pyi)']
['(stdlib/3/socketserver.pyi)', '(stdlib/2/SocketServer.pyi)']
['(stdlib/3/encodings/utf_8.pyi)', '(stdlib/2/encodings/utf_8.pyi)']
['(stdlib/3/encodings/__init__.pyi)', '(stdlib/2/encodings/__init__.pyi)']
['(stdlib/3.7/dataclasses.pyi)', '(third_party/3/dataclasses.pyi)']
['(third_party/2and3/backports/ssl_match_hostname.pyi)', '(third_party/2and3/requests/packages/urllib3/packages/ssl_match_hostname/_implementation.pyi)']
['(stdlib/2and3/threading.pyi)', '(stdlib/2and3/_dummy_threading.pyi)']
['(stdlib/2and3/xml/dom/pulldom.pyi)', '(stdlib/2and3/xml/dom/minidom.pyi)']
['(third_party/2and3/cryptography/hazmat/primitives/asymmetric/__init__.pyi)', '(third_party/2and3/cryptography/hazmat/primitives/__init__.pyi)', '(third_party/2and3/cryptography/hazmat/__init__.pyi)']
['(third_party/2and3/markupsafe/_native.pyi)', '(third_party/2and3/markupsafe/_speedups.pyi)']
['(third_party/3/six/moves/email_mime_multipart.pyi)', '(third_party/2/six/moves/email_mime_multipart.pyi)']
['(third_party/3/six/moves/email_mime_base.pyi)', '(third_party/2/six/moves/email_mime_base.pyi)']
['(third_party/3/six/moves/BaseHTTPServer.pyi)', '(third_party/3/six/moves/SimpleHTTPServer.pyi)', '(third_party/3/six/moves/CGIHTTPServer.pyi)']
['(third_party/3/six/moves/urllib_response.pyi)', '(third_party/2/six/moves/urllib_response.pyi)']
['(third_party/3/six/moves/urllib_request.pyi)', '(third_party/2/six/moves/urllib_request.pyi)']
['(third_party/3/six/moves/tkinter_tkfiledialog.pyi)', '(third_party/3/six/moves/tkinter_filedialog.pyi)']
['(third_party/3/six/moves/email_mime_nonmultipart.pyi)', '(third_party/2/six/moves/email_mime_nonmultipart.pyi)']
['(third_party/3/six/moves/urllib/__init__.pyi)', '(third_party/2/six/moves/urllib/__init__.pyi)']
['(third_party/3/docutils/__init__.pyi)', '(third_party/3/docutils/parsers/__init__.pyi)', '(third_party/3/docutils/parsers/rst/nodes.pyi)', '(third_party/3/docutils/parsers/rst/__init__.pyi)']
['(third_party/3/contextvars.pyi)', '(stdlib/3.7/contextvars.pyi)']

  12. JelleZijlstra commented on Jun 11, 2020

    @JelleZijlstra
    JelleZijlstra
    on Jun 11, 2020
    Member

    I agree, this hardly seems worth it any more. Hopefully in the not-too-distant future we can drop support for Python 2 completely.

  13. hauntsaninja commented on Jun 11, 2020

    @hauntsaninja
    hauntsaninja
    on Jun 11, 2020
    Collaborator

    Okay, sounds good! Closing this and the similar #1427

  14. hauntsaninja
    closed this as completedon Jun 11, 2020
  15. hswong3i
    added a commit that references this issue on May 25, 2025

    fix CI (python#4)

    447b62c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    stubs: improvementImprove/refactor existing annotations, other stubs issues

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions