Skip to content

Merge Prism parser binding and compiler into core - #9686

Draft
headius wants to merge 1 commit into
jruby:masterfrom
headius:merge_prism_parser
Draft

headius wants to merge 1 commit into
jruby:masterfrom
headius:merge_prism_parser

Conversation

@headius

@headius headius commented Sep 15, 2026

Copy link
Copy Markdown
Member

This library lived previously as the jruby-prism artifact, during a time when we expected to be updating its code independently of JRuby proper. This was also done at a time when we still depended on the native library build of Prism, which introduced various challenges to shipping a cross-platform artifact.

Updates over the last year have brought the parser and compiler more closely in line with JRuby's built-in legacy parser, and the use of new WASM AOT features from Chicory Redline we can now include the parser in JRuby out of the box. Having this be a separate project and artifact just gets in the way of us fully adopting Prism.

This patch moves the contents of jruby-prism into JRuby core.

Note that loading the current WASM AOT version of the Prism parser impacts performance by about 30%. Before it can be fully enabled we need to address this startup hit along with the remaining language compatibility issues.

@headius headius added this to the JRuby 10.1.x milestone Sep 15, 2026
@headius

headius commented Sep 15, 2026

Copy link
Copy Markdown
Member Author

JRuby startup times for -e 1 with legacy and prism parsers:

[] jruby $ jruby --rmcache -e 1
[] jruby $ time jruby -e 1          
jruby -e 1  2.16s user 0.10s system 107% cpu 2.098 total
[] jruby $ time jruby -e 1
jruby -e 1  1.35s user 0.08s system 96% cpu 1.478 total
[] jruby $ time jruby -e 1
jruby -e 1  1.34s user 0.07s system 90% cpu 1.553 total
[] jruby $ time jruby -e 1
jruby -e 1  1.32s user 0.07s system 85% cpu 1.629 total
[] jruby $ time jruby --dev -e 1
jruby --dev -e 1  0.68s user 0.06s system 51% cpu 1.431 total
[] jruby $ time jruby --dev -e 1
jruby --dev -e 1  0.68s user 0.06s system 51% cpu 1.452 total
[] jruby $ time jruby --dev -e 1
jruby --dev -e 1  0.68s user 0.06s system 53% cpu 1.388 total
[] jruby $ jruby --rmcache               
[] jruby $ time jruby -Xparser.prism -e 1
jruby -Xparser.prism -e 1  3.94s user 0.19s system 156% cpu 2.634 total
[] jruby $ time jruby -Xparser.prism -e 1
jruby -Xparser.prism -e 1  2.77s user 0.20s system 141% cpu 2.094 total
[] jruby $ time jruby -Xparser.prism -e 1
jruby -Xparser.prism -e 1  2.80s user 0.20s system 136% cpu 2.204 total
[] jruby $ time jruby -Xparser.prism -e 1
jruby -Xparser.prism -e 1  2.74s user 0.17s system 140% cpu 2.080 total
[] jruby $ time jruby --dev -Xparser.prism -e 1
jruby --dev -Xparser.prism -e 1  1.41s user 0.15s system 87% cpu 1.788 total
[] jruby $ time jruby --dev -Xparser.prism -e 1
jruby --dev -Xparser.prism -e 1  1.40s user 0.15s system 84% cpu 1.845 total
[] jruby $ time jruby --dev -Xparser.prism -e 1
jruby --dev -Xparser.prism -e 1  1.39s user 0.15s system 89% cpu 1.719 total

Much of the slowdown does appear to be related to loading the Chicory WASM subsystem. Here are the biggest delays during a cached (AppCDS) -e 1 run with JVM flag -verbose:class:

This hit seems to be common to both modes and probably relates to the loading of the jffi shared library:

[0.244s][info][class,load] java.nio.channels.Channels$ReadableByteChannelImpl source: shared objects file (top)
[0.245s][info][class,load] java.io.FileOutputStream$1 source: shared objects file (top)
[1.986s][info][class,load] com.kenai.jffi.internal.StubLoader$CPU source: shared objects file (top)
[1.988s][info][class,load] java.lang.invoke.LambdaForm$MH/0x00000c0004044c00 source: __JVM_LookupDefineClass__
[1.988s][info][class,load] java.lang.invoke.LambdaForm$MH/0x00000c0004045000 source: __JVM_LookupDefineClass__

There are a few jumps here while loading Chicory internals needed to parse the WASM and connect to Redline:

[2.024s][info][class,load] com.jruby.internal.org.dylibso.chicory.wasm.Parser$$Lambda/0x000000fe010d3778 source: shared objects file (top)
[2.025s][info][class,load] com.jruby.internal.org.dylibso.chicory.wasm.Parser$$Lambda/0x000000fe010d39b8 source: shared objects file (top)
[2.105s][info][class,load] com.jruby.internal.org.dylibso.chicory.wasm.Validator$$Lambda/0x000000fe010b5280 source: shared objects file (top)
[2.105s][info][class,load] java.util.Arrays$ArrayItr source: shared objects file
[2.193s][info][class,load] com.jruby.internal.io.roastedroot.redline.bridge.RedlineBridge$$Lambda/0x000000fe010d6000 source: shared objects file (top)
[2.193s][info][class,load] com.jruby.internal.org.dylibso.chicory.runtime.ByteBufferMemory$$Lambda/0x000000fe010d6248 source: shared objects file (top)

The CompiledTrampolines class is consistently around 200ms to load:

[2.253s][info][class,load] java.lang.foreign.MemorySegment$Scope source: shared objects file (top)
[2.253s][info][class,load] jdk.internal.foreign.MemorySessionImpl source: shared objects file (top)
[2.449s][info][class,load] com.jruby.internal.io.roastedroot.redline.bridge.RedlineBridge$CompiledTrampolines source: shared objects file (top)
[2.456s][info][class,load] java.lang.invoke.MethodHandle$1 source: shared objects file (top)

After this point, things seem to be up and running and the remaining class loads happen with fairly little delay. Total time for this run was 2.950s.

I suspect we are not getting the full benefit of AppCDS for some of the Chicory AOT compiled stuff, perhaps because it's too large. This was tested on JDK 25.

cc @andreaTP FYI

@headius

headius commented Sep 15, 2026

Copy link
Copy Markdown
Member Author

This merge can wait until the Endive migration for the WASM Prism has been released, but it's not a prerequisite (none of the code in JRuby directly references Chicory).

See ruby/prism#4229

@headius

headius commented Sep 15, 2026

Copy link
Copy Markdown
Member Author

Kinda feels like this should be under org.jruby.parser.prism and what's in org.jruby.parser should move to org.jruby.parser.legacy or similar. cc @enebo for thoughts

@headius

headius commented Sep 15, 2026 •

Copy link
Copy Markdown
Member Author

I have discovered and am in the process of fixing a JFFI bug causing it to always write out a new dynamic library file each time on MacOS, which greatly improves startup time.

The increase from loading Prism remains very high, however:

(both times after warming up OS and JVM caches)

[] jruby $ time jruby -e 1
jruby -e 1  1.25s user 0.06s system 225% cpu 0.578 total
[] jruby $ time jruby -Xparser.prism -e 1
jruby -Xparser.prism -e 1  2.89s user 0.14s system 253% cpu 1.195 total

This library lived previously as the jruby-prism artifact, during
a time when we expected to be updating its code independently of
JRuby proper. This was also done at a time when we still depended
on the native library build of Prism, which introduced various
challenges to shipping a cross-platform artifact.

Updates over the last year have brought the parser and compiler
more closely in line with JRuby's built-in legacy parser, and the
use of new WASM AOT features from Chicory Redline we can now
include the parser in JRuby out of the box. Having this be a
separate project and artifact just gets in the way of us fully
adopting Prism.

This patch moves the contents of jruby-prism into JRuby core.

Note that loading the current WASM AOT version of the Prism parser
impacts performance by about 30%. Before it can be fully enabled
we need to address this startup hit along with the remaining
language compatibility issues.
@headius

headius commented Sep 15, 2026

Copy link
Copy Markdown
Member Author

Updated to use 0.0.5-SNAPSHOT of the prism parser artifact, which moves to Endive.

@andreaTP

Copy link
Copy Markdown
Contributor

com.jruby.internal.org.dylibso.chicory.wasm.Parser

This is suspicious, do you have a direct link to where the Wasm Instance is being created?

@andreaTP

Copy link
Copy Markdown
Contributor

@headius I was testing redline on QuickJs and found a few things to be improved, let me iterate and get back when I publish the next version.

@headius

headius commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

@andreaTP The actual code for this still lives in the ruby/prism project so that's the place to look. This PR just eliminates a middleman project we were using to isolate JRuby from the native library.

I haven't had a chance to get the new endive version of prism released.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants