➜
See More

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author ncoghlan
Recipients meador.inge, michael.foord, ncoghlan
Date 2010-09-28.21:54:16
SpamBayes Score 0.00853855
Marked as misclassified No
Message-id <[email protected]>
In-reply-to
Content 
Possible approach (untested):

def get_tokens(source):
    if hasattr(source, "encode"):
        # Already decoded, so bypass encoding detection
        return _tokenize(io.StringIO(source).readline, None)
    # Otherwise attempt to detect the correct encoding
    return tokenize(io.BytesIO(source).readline)
History
Date User Action Args
2010-09-28 21:54:18ncoghlansetrecipients: + ncoghlan, michael.foord, meador.inge
2010-09-28 21:54:18ncoghlansetmessageid: <[email protected]>
2010-09-28 21:54:16ncoghlanlinkissue9969 messages
2010-09-28 21:54:16ncoghlancreate