➜
See More

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author Joshua.Landau
Recipients Joshua.Landau
Date 2016-04-25.01:58:43
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <[email protected]>
In-reply-to
Content 
This is effectively a continuation of https://bugs.python.org/issue9712.

The line in Lib/tokenize.py

    Name = r'\w+'

must be changed to a regular expression that accepts Other_ID_Start at the start and Other_ID_Continue elsewhere. Hence tokenize does not accept '℘·'.


See the reference here:

    https://docs.python.org/3.5/reference/lexical_analysis.html#identifiers

I'm unsure whether unicode normalization (aka the `xid` properties) needs to be dealt with too.


Credit to toriningen from http://stackoverflow.com/a/29586366/1763356.
History
Date User Action Args
2016-04-25 01:58:44Joshua.Landausetrecipients: + Joshua.Landau
2016-04-25 01:58:44Joshua.Landausetmessageid: <[email protected]>
2016-04-25 01:58:44Joshua.Landaulinkissue26843 messages
2016-04-25 01:58:43Joshua.Landaucreate