Repository navigation
Add "java modified utf-8" codec #47106
Description
| BPO | 2857 |
|---|---|
| Nosy | @malemburg, @loewis, @birkenfeld, @abalkin, @vstinner, @ezio-melotti, @serhiy-storchaka |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2012-04-24.12:26:37.040>
created_at = <Date 2008-05-15.03:08:38.194>
labels = ['type-feature', 'library', 'expert-unicode']
title = 'Add "java modified utf-8" codec'
updated_at = <Date 2012-04-24.12:26:37.038>
user = 'https://bugs.python.org/phr'bugs.python.org fields:
activity = <Date 2012-04-24.12:26:37.038>
actor = 'loewis'
assignee = 'none'
closed = True
closed_date = <Date 2012-04-24.12:26:37.040>
closer = 'loewis'
components = ['Library (Lib)', 'Unicode']
creation = <Date 2008-05-15.03:08:38.194>
creator = 'phr'
dependencies = []
files = ['21965']
hgrepos = []
issue_num = 2857
keywords = ['patch']
message_count = 26.0
messages = ['66843', '66852', '66854', '66855', '66857', '66862', '66866', '67368', '123484', '123770', '135757', '135772', '135776', '135796', '135797', '141938', '141940', '141949', '141955', '141956', '141957', '142017', '159130', '159133', '159136', '159137']
nosy_count = 10.0
nosy_names = ['lemburg', 'loewis', 'georg.brandl', 'phr', 'belopolsky', 'moese', 'vstinner', 'ezio.melotti', 'tchrist', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'test needed'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue2857'
versions = ['Python 3.3']
Activity
phr commentedon May 15, 2008
For object serialization and some other purposes, Java encodes unicode
strings with a modified version of utf-8:
http://en.wikipedia.org/wiki/UTF-8#Java
http://java.sun.com/javase/6/docs/api/java/io/DataInput.html#modified-utf-8
It is used in Lucene index files among other places.
It would be useful if Python had a codec for this, maybe called "UTF-8J"
or something like that.
malemburg commentedon May 15, 2008
What would you use such a codec for ?
From the references you gave, it is only used internally for Java object
serialization, so wouldn't really be of much use in Python.
[-]add coded for java modified utf-8[/-][+]add codec for java modified utf-8[/+]phr commentedon May 15, 2008
Some java applications use it externally. The purpose seems to be to
prevent NUL bytes from appearing inside encoded strings which can
confuse C libraries that expect NUL's to terminate strings. My
immediate application is parsing lucene indexes:
http://lucene.apache.org/java/docs/fileformats.html#Chars
phr commentedon May 15, 2008
Also, according to wikipedia, tcl also uses that encoding.
malemburg commentedon May 15, 2008
TCL only uses the codec for internal represenation. You might want to
interface to TCL at the C level and use the codec there, but is that
really a good reason to include the codec in the Python stdlib ?
Dito for parsing Lucene indexes.
I think you're better off writing your own codec and registering it with
the Python codec registry at application start-up time.
phr commentedon May 15, 2008
I'm not sure what you mean by "ditto for Lucene indexes". I wasn't
planning to use C code. I was hoping to write Python code to parse
those indexes, then found they use this weird encoding, and Python's
codec set is fairly inclusive already, so this codec sounded like a
reasonably useful addition. It probably shows up other places as well.
It might even be a reasonable internal representation for Python, which
as I understand it currently can't represent codepoints outside the BMP.
Also, it is used in Java serialization, which I think of as a somewhat
weird and whacky thing, but it's conceivable that somebody someday might
want to write a Python program that speaks the Java serialization
protocol (I don't have a good sense of whether that's feasible).
Writing an application specific codec with the C API is doable in
principle, but it seems like an awful lot of effort for just one quickie
program. These indexes are very large and so writing the codec in
Python would probably be painfully slow.
birkenfeld commentedon May 15, 2008
Since we also support oddball codecs like UTF-8-SIG, why not this one too?
Given the importance of UTF-8, it seems a good idea to support common
variations.
malemburg commentedon May 26, 2008
Ok, if you can write a patch implementing the codec, we'll add it.
Please use the name "utf-8-java" and codec name utf_8_java.py.
abalkin commentedon Dec 6, 2010
I wonder if tkinter should use this encoding.
vstinner commentedon Dec 11, 2010
Tkinter is used to build graphical interfaces. I don't think that users write nul bytes with their keyboard. But there is maybe a use case?
moese commentedon May 11, 2011
I use the hachoir Python package to parse Java .class files and extract the strings from them and having support for Java modified UTF-8 would have been nice.
4 remaining items
tchrist commentedon Aug 12, 2011
Please do not call this "utf-8-java". It is called "cesu-8" per UTS#18 at:
http://unicode.org/reports/tr26/
CESU-8 is *not* a a valid Unicode Transform Format and should not be called UTF-8. It is a real pain in the butt, caused by people who misunderand Unicode mis-encoding UCS-2 into UTF-8, screwing it up. I understand the need to be able to read it, but call it what it is, please.
Despite the talk about Lucene, I note that the Perl port of Lucene uses real UTF-8, not CESU-8.
birkenfeld commentedon Aug 12, 2011
+1 for calling it by the correct name (the docs can of course state that this is equivalent to "Java Modified UTF-8" or however they like to call it).
malemburg commentedon Aug 12, 2011
Tom Christiansen wrote:
CESU-8 is a different encoding than the one we are talking about.
The only difference between UTF-8 and the modified one is the different
encoding for the U+0000 code point to have the output not contain
any NUL bytes.
[-]add codec for java modified utf-8[/-][+]Add CESU-8 codec ("java modified utf-8")[/+]malemburg commentedon Aug 12, 2011
Corrected the title again. See my comment.
[-]Add CESU-8 codec ("java modified utf-8")[/-][+]Add "java modified utf-8" codec[/+]malemburg commentedon Aug 12, 2011
Marc-Andre Lemburg wrote:
Please open a new ticket, if you want to add a CESU-8 codec.
Looking at the relevant use cases, I'm at most +0 on adding the
modified UTF-8 codec. I think such codecs can well live outside
the stdlib on PyPI.
moese commentedon Aug 12, 2011
Python does have other "weird" encodings like bz2 or rot13.
Beside, batteries included :)
vstinner commentedon Aug 13, 2011
No, it has no more such weird encodings.
serhiy-storchaka commentedon Apr 24, 2012
As far as I understand, this codec can be implemented in Python. There is no need to modify the interpreter core.
loewis commentedon Apr 24, 2012
Serhiy: your functions to not constitute a Python codec. For example, there is no support for error handlers in them.
serhiy-storchaka commentedon Apr 24, 2012
Yes, it is not a codec in Python library terminology. It's just a pair
of functions, the COder and DECoder, which is enough for the task of
hacking Java serialized data. I don't think that such specific task
leads to the change of the interpreter core.
However, translators that convert the non-BMP characters to a surrogate
pair and back, would be useful in the standard library. They need to
work with a non-standard encodings (CESU-8, MUTF-8, cp65001, some
Tk/IDLE issues). This is a fairly common task.
loewis commentedon Apr 24, 2012
Ok, I'm closing this entire issue as "won't fix", then. There apparently is a need for functionality like this, but there is apparently also a concern that this is too specialized for the standard library.
As it is possible to implement this as a stand-alone library, I encourage interested users to design a package for PyPI that has this functionality collected for reuse. If the library is then widely used after some time, this issue can be reconsidered.