Skip to content

Consider encoding in ByteList equality - #9646

Open
headius wants to merge 1 commit into
jruby:masterfrom
headius:bytelist_equals_encoding
Open

Consider encoding in ByteList equality#9646
headius wants to merge 1 commit into
jruby:masterfrom
headius:bytelist_equals_encoding

Conversation

@headius

@headius headius commented Sep 4, 2026

Copy link
Copy Markdown
Member

Two ByteLists may have exactly the same bytes, but not the same encoding. Such ByteLists have been treated as equals() but we have frequently run into bugs from this behavior. This PR finally adds encoding to the conditions required for ByteList equality.

@headius
Consider encoding in ByteList equality
9331925 
Two ByteLists may have exactly the same bytes, but not the same
encoding. Such ByteLists have been treated as equals() but we have
frequently run into bugs from this behavior. This PR finally adds
encoding to the conditions required for ByteList equality.
@headius headius added this to the JRuby 10.1.2.0 milestone Sep 4, 2026
@headius

headius commented Sep 4, 2026

Copy link
Copy Markdown
Member Author

Note that the caching in #9644 depends on true ByteList equality, so this will probably be required to ship that PR as it is currently implemented.

@headius

headius commented Sep 4, 2026

Copy link
Copy Markdown
Member Author

A good experiment but we appear to have many places that depend on ByteList equality only considering the contents of the byte array.

It fails at startup, most likely because two versions of the ByteList for the import_classes variable in the code below get confused, and we don't see that it's a variable when compiling the access.

core/src/main/ruby/jruby/java/core_ext/object.rb:

# @see Module.java_import
def java_import(*import_classes, &block)
  Object.send :java_import, *import_classes, &block
end

Resulting error:

$ jruby -e 1
NameError: undefined local variable or method 'import_classes' for main
  <main> at uri:classloader:/jruby/java/core_ext/object.rb:2
    load at org/jruby/RubyKernel.java:1403
  <main> at uri:classloader:/jruby/java/core_ext.rb:4
    load at org/jruby/RubyKernel.java:1403
  <main> at file:///Users/headius/work/jruby/lib/jruby.jar!/jruby/java.rb:37

Likely source of the problem:

jruby/core/src/main/java/org/jruby/parser/RubyParserBase.java

Lines 1332 to 1340 in 6cdb065

public RubySymbol symbolID(ByteList identifierValue) {
// FIXME: We walk this during identifier construction so we should calculate CR without having to walk twice.
if (RubyString.scanForCodeRange(identifierValue) == StringSupport.CR_BROKEN) {
Ruby runtime = getRuntime();
throw runtime.newSyntaxError(str(runtime, "invalid symbol in encoding " + lexer.getEncoding() + " :\"", inspectIdentifierByteList(runtime, identifierValue), "\""), getFile());
}
return RubySymbol.newIDSymbol(getRuntime(), identifierValue);
}
See More

The symbol table is keyed on ByteList, and the code parsed here may be producing two bytelists with the same content but different encodings (US-ASCII vs UTF-8 for example).

@headius headius removed this from the JRuby 10.1.2.0 milestone Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@headius