Skip to content

Create PyUnicode_EncodeFSDefault() function #52961

Closed
Closed
Create PyUnicode_EncodeFSDefault() function#52961
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)topic-unicode

Description

@vstinner
vstinner
opened on May 14, 2010
BPO 8715
Nosy @malemburg, @loewis, @gpshead, @pitrou, @vstinner, @benjaminp, @ezio-melotti
Files
  • pyunicode_encodefsdefault-3.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2010-05-15.16:28:41.394>
created_at = <Date 2010-05-14.16:53:52.192>
labels = ['interpreter-core', 'expert-unicode']
title = 'Create PyUnicode_EncodeFSDefault() function'
updated_at = <Date 2010-05-15.16:28:41.393>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2010-05-15.16:28:41.393>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2010-05-15.16:28:41.394>
closer = 'vstinner'
components = ['Interpreter Core', 'Unicode']
creation = <Date 2010-05-14.16:53:52.192>
creator = 'vstinner'
dependencies = []
files = ['17341']
hgrepos = []
issue_num = 8715
keywords = ['patch']
message_count = 6.0
messages = ['105721', '105722', '105779', '105809', '105810', '105819']
nosy_count = 8.0
nosy_names = ['lemburg', 'loewis', 'gregory.p.smith', 'pitrou', 'vstinner', 'benjamin.peterson', 'ezio.melotti', 'Arfrever']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue8715'
versions = ['Python 3.2']

    Activity

    1. vstinner commented on May 14, 2010

      @vstinner
      vstinner
      on May 14, 2010
      MemberAuthor

      PyUnicode_EncodeFSDefault() is the opposite of PyUnicode_DecodeFSDefault(AndSize)() and is similar to the new function os.fsencode(). As you can see in the patch, it simplifies many functions.

      /* Encodes a Unicode object to Py_FileSystemDefaultEncoding with the
      "surrogateescape" error handler and returns a bytes object.

      If Py_FileSystemDefaultEncoding is not set, fall back to UTF-8.
      */

      PyAPI_FUNC(PyObject*) PyUnicode_EncodeFSDefault(
    PyObject *unicode
    );

      The function unify the behaviour when Py_FileSystemDefaultEncoding is NULL: use UTF-8 whereas import uses ASCII. Other functions did already fall back to UTF-8: PyUnicode_AsEncodedString() uses PyUnicode_GetDefaultEncoding() (hardcoded to utf8 in Python3) if encoding is NULL

      The patch does also fix tkinter module initializer (use surrogateescape error handler, instead of strict).

      The patch was first attached to issue bpo-8611.

    2. vstinner
      added
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      topic-unicode
      on May 14, 2010

    3. vstinner commented on May 14, 2010

      @vstinner
      vstinner
      on May 14, 2010
      MemberAuthor

      Ooops, I attached the wrong version of the patch. Version 3 changes the documentation (Encodes => Encode).

    4. vstinner commented on May 15, 2010

      @vstinner
      vstinner
      on May 15, 2010
      MemberAuthor

      Notes for myself:

      • "Encodes" and "fallback" in .h documentation => "Encode", "fall back"
      • bootstrap failure on Windows: import did use default error handler, it uses surrogateescape error handler, but PyUnicode_EncodeString() doesn't have codec "fast-path" for MBCS+surrogateescape.

    5. vstinner commented on May 15, 2010

      @vstinner
      vstinner
      on May 15, 2010
      MemberAuthor

      bootstrap failure on Windows: import did use default error handler,
      it uses surrogateescape error handler, but PyUnicode_EncodeString()
      doesn't have codec "fast-path" for MBCS+surrogateescape.

      I enabled "shortcuts" in PyUnicode_EncodeString() for any error handler (not only the default error handler, strict) in r81192.

    6. vstinner commented on May 15, 2010

      @vstinner
      vstinner
      on May 15, 2010
      MemberAuthor

      PyUnicode_AsEncodedString() contains a special path for the file system encoding. I don't think that it is still needed, but I don't know how to check that.

      /* During bootstrap, we may need to find the encodings
   package, to load the file system encoding, and require the
   file system encoding in order to load the encodings
   package.

   Break out of this dependency by assuming that the path to
   the encodings module is ASCII-only.  XXX could try wcstombs
   instead, if the file system encoding is the locale's
   encoding. */
else if (Py_FileSystemDefaultEncoding &&
         strcmp(encoding, Py_FileSystemDefaultEncoding) == 0 &&
         !PyThreadState_GET()->interp->codecs_initialized)
    return PyUnicode_EncodeASCII(PyUnicode_AS_UNICODE(unicode),
                                 PyUnicode_GET_SIZE(unicode),
                                 errors);

    7. vstinner commented on May 15, 2010

      @vstinner
      vstinner
      on May 15, 2010
      MemberAuthor

      Commited as r81194 (py3k), blocked in 3.1 (r81195).

    8. vstinner
      closed this as completedon May 15, 2010
    9. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      interpreter-core(Objects, Python, Grammar, and Parser dirs)topic-unicode

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions