Skip to content

Implement method coverage - #9676

Open
sferik wants to merge 19 commits into
jruby:masterfrom
sferik:method-coverage
Open

sferik wants to merge 19 commits into
jruby:masterfrom
sferik:method-coverage

Conversation

@sferik

@sferik sferik commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

This adds the methods mode of the Coverage library. Results have the same shape, keys and counts as CRuby:

Coverage.start(methods: true)
require "foo"
Coverage.result
# => { "/path/foo.rb" => { methods: { [Foo, :bar, 3, 2, 5, 5] => 2, ... } } }

Keys are [owner, name, start_line, start_column, end_line, end_column], exactly as CRuby reports them (1-based lines, 0-based byte columns, def through end, or the block/lambda passed to define_method).

Addresses the method half of #5147. Once this is merged, I intend to work on branch coverage, which I think would be substantially trickier.

MethodCoverage is the JRuby analogue of the per-method-entry counters CRuby keeps (its me2counter hash). It is created in RubyModule#putMethod, the one place every method entry lands, whenever methods are being measured and the entry's source file is being tracked, and it is attached to the DynamicMethod. As in CRuby, entries that only forward to another entry get no counter of their own: aliases, private :inherited_method in a subclass, define_method(:x, instance_method(:aliased)). Calls through them count toward the original. module_function, Class#dup and define_method(name, method_object) create fresh entries with the new owner, again as in CRuby.

CRuby counts on the CALL event, which fires after the arguments have been bound, so a call that fails with an ArgumentError (wrong arity, missing keyword) or a raising default value is not counted. To match that, the count happens inside the body rather than in DynamicMethod#call:

  • every Ruby-level call path (InterpretedIRMethod, MixedModeIRMethod, CompiledIRMethod, ProcMethod) hands the entry's counter to the body through a slot on ThreadContext
  • ReceiveMethodCoverageInstr, the first instruction of the body, takes the counter into a temp before any argument is received, so nested calls made while receiving arguments (default value expressions, to_ary conversions) cannot take it
  • CoverMethodInstr, emitted right after argument receipt, increments it. The counter records which scope it belongs to, so a stale hand-off can never be attributed to another method.
  • a define_method method whose lambda arity check fails (that check runs in Java, before the body) clears the counter it left pending, so a later plain call of the same proc is not counted as a call of the method
  • the initialize of a Java subclass, which ConcreteJavaProxy runs through the split-constructor machinery rather than DynamicMethod#call (or skips entirely when it is a plain forwarding super), gets the hand-off in startSplitSuperCall and is counted directly when its body is skipped

Blocks get the same two probes because any block can become a method via define_method. When called as plain blocks they see no counter and the probes are a null check. Invokedynamic call sites no longer bind directly to the compiled handle of a method that is being counted, so the hand-off is never bypassed.

Counting is a single lock-free atomic add (VarHandle#getAndAdd), so threads calling the same method neither serialize nor lose increments. A new test verifies 8 threads × 5000 calls produce exactly 40000. Registration and result conversion are synchronized on CoverageData as the existing line coverage code already is.

The grammar now records byte columns and end positions for def, blocks and lambdas (DefNode/IterNode gained getStartColumn/getEndColumn, and MethodDefNode#getEndLine is now the line of the end keyword rather than the end of the body). Lambda spans follow CRuby: from the start of the parameter list, or just past -> when there is none, through the end of the body. Both generated parsers were regenerated from RubyParser.y with jay; the toolchain was verified to reproduce the checked-in sources byte-for-byte before the grammar was touched. The spans are carried into IRScope and survive the block-to-method conversion used by define_method.

CoverageData now keeps a FileCoverage per file (lines, methods, and room for branches) instead of a bare IntList. Results contain only the requested modes, branches reports {} for now, files are listed in parse order like CRuby, and Coverage.supported?(:methods) is true.

Two other notes:

  • IRClosure silently dropped the coverageMode passed to its constructor, so blocks converted into methods for define_method were built without any coverage instrumentation.
  • oneshot_lines probes disarmed themselves when hit while coverage was suspended, losing the line for good. They now stay armed until the line is actually counted.

Known remaining differences, both pre-existing and unrelated to counting: JRuby keys loaded files by their real path, and Class#dup keeps aliases as aliases (CRuby materializes them into real entries). Prism builds do not supply columns yet, so keys would carry -1 columns there.

Following @enebo's note in #5147 (comment), branches will be marked at IR build time where the builder knows it is compiling a Ruby-semantic branch, rather than recovered from the IR's own jumps. The per-file container, the mode-driven result conversion and the parser source spans this PR adds are the groundwork for that.


@Override
public IRubyObject call(ThreadContext context, IRubyObject self, RubyModule clazz, String name, IRubyObject[] args, Block block) {
prepareMethodCoverage(context);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this approach the best we can do, having to invoke prepareMethodCoverage from all places?

not sure how much thought went into the PR but would be nice to consider other options...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did weigh alternatives but open to suggestions I may not have considered.

The counter has to be per method entry (CRuby keys results by [owner, name, location], so Class.new { def x; end } in a loop or %w[a b].each { |n| define_method(n) { } } produce distinct counts from one scope), and the count has to happen after arguments are bound (CRuby counts on the CALL event, so a call that dies on arity or a missing keyword is not counted). So something has to get the entry's counter from the DynamicMethod into the body, and the body has to do the increment.

Here are the options I considered:

  1. Wrap the entry in a delegating DynamicMethod at registration, the way ProfilingDynamicMethod/MethodEnhancer do it, so the hand-off lives in one class. The problem is that a wrapper changes the concrete class of the entry and several places key real behavior off that: InvokeSite decides keyword handling by entry.method instanceof AbstractIRMethod, refinement import_methods raises for non-AbstractIRMethod entries, Method#== for define_method methods checks instanceof ProcMethod, ruby2_keywords reads the static scope through the same check, and the Java proxy code detects user-defined initialize/new with it. Each would need an unwrap. That is more invasive and more fragile than one line per call variant, and it only matters while coverage is on.
  2. I didn’t count in call without a body probe because it’s the same number of touch points and wrong for calls that fail on their arguments.
  3. Hook the trace CALL event won’t work because it’s only emitted in full-trace mode and before arguments are received.
  4. I didn’t fold the hand-off into the compiled MethodHandles because it covers CompiledIRMethod only. Interpreted and mixed-mode paths still need explicit hooks.
  5. I didn’t pass the entry into the body because it touches the interpreter, JIT, and indy signatures.

The one-liner per arity variant follows what these classes already do for every per-call concern: doDebug() and the callCount/tryJit check are repeated in each call of MixedModeIRMethod and InterpretedIRMethod. prepareMethodCoverage sits next to those. With coverage off it is a field load plus a predictable null check, which is cheaper than the callCount check beside it.

If you would rather see fewer sites, InterpretedIRMethod's ten variants could route through the five Interpreter.INTERPRET_METHOD helpers by passing the method instead of its scope, but that trades ten trivial lines for a signature change on a public helper. Happy to do that if you think it is worth it!

Comment thread core/src/main/java/org/jruby/internal/runtime/methods/MixedModeIRMethod.java Outdated
Comment thread core/src/main/java/org/jruby/ir/runtime/IRRuntimeHelpers.java
Comment thread core/src/main/java/org/jruby/ext/coverage/FileCoverage.java Outdated
@headius

headius commented Sep 14, 2026

Copy link
Copy Markdown
Member

@sferik thank you for jumping in here and taking a shot at this! I agree with your analysis of how CRuby does this and I'm sure we can work through refining your patch to minimize performance impact. I'm going to mark this as 10.1.x so we can take a little more time to make it perfect.

These missing coverage features have been bothering me more lately so I'm excited to get some help implementing them!

@headius headius modified the milestones: JRuby 10.1.2.0, JRuby 10.0.x Sep 14, 2026
@sferik

sferik commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

@headius Thanks Charlie! I’m happy to help move these coverage features forward however I can, whether or not you actually use any of this code. I’ve added a bunch of features to SimpleCov in the past few releases, and I’d like for JRuby users to benefit from those changes. More selfishly, I’d like to remove the unless RUBY_ENGINE == "jruby" conditions from that codebase.

My experience with Java and the JVM is limited, so I’ll happily take whatever advice you have. I’m working through @kares feedback now.

Targeting 10.1 sounds good to me. These features have been missing from JRuby for a while, so there’s no particular urgency and I think it’s more important that this is done right than done quickly. I’m particularly sensitive to any performance impact it might have on production code. If those impacts can’t be mitigated, I’m open to gating them behind --debug (or a similar flag), since users typically only use these features in test environments. That said, SimpleCov 1.3 introduced a production mode for finding dead code, so I’d want to exclude oneshot line coverage from that flag, if we end up going that route. Hopefully we can optimize everything enough that we don’t have to!

@sferik

sferik commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

@headius I just tried rebasing this branch from 10.1-dev but that looks like it hasn’t been updated since February and is an ancestor of master, so I just rebased from master to resolve a conflict. Please advise if there’s something you’d like me to do differently.

Coverage's methods mode now works and produces the same results as CRuby:

  { file => { methods: { [owner, name, start_line, start_column,
                          end_line, end_column] => call_count } } }

Design follows CRuby's per-method-entry counters and JaCoCo-style probes:

* A MethodCoverage counter is created for every method entry (each def
  or define_method landing in a module, in RubyModule#putMethod) whose
  source file is being tracked, and attached to the DynamicMethod.
  Aliases, visibility changes of inherited methods and other forwarding
  entries get no counter of their own. Their calls count toward the
  original, as in CRuby.
* Every Ruby-level call path (interpreted, mixed-mode, JIT-compiled, and
  proc-based define_method) hands the counter to the body through the
  ThreadContext. The body takes it in its first instruction
  (ReceiveMethodCoverageInstr) and counts the call once its arguments
  have been received (CoverMethodInstr), so calls that fail on arity,
  keywords or a raising default value are not counted, matching where
  CRuby fires the CALL event. Blocks get the same probes since any block
  may become a method via define_method. Indy call sites no longer bind
  directly to the compiled handle of a method that is being counted.
* Counting is a lock-free atomic add, so threads calling the same method
  in parallel neither serialize nor lose increments. Entries sharing a
  key are summed like CRuby.
* The parser now records byte columns and end positions of def, block
  and lambda definitions (DefNode/IterNode source spans), carried into
  IRScope. Both generated parsers were regenerated from RubyParser.y
  with jay.

Coverage data is now kept per file in FileCoverage (lines, methods, and
room for branches), results only contain the requested modes (branches
still reports an empty hash), files are listed in parse order, and
Coverage.supported?(:methods) is true.

Also fixes two pre-existing issues this work surfaced: IRClosure dropped
the coverageMode it was given, so blocks converted into methods for
define_method were built without any coverage and oneshot_lines probes
disarmed themselves when hit while coverage was suspended.

Un-excludes the CRuby method coverage tests and the :all-modes spec, and
adds JRuby-specific tests for parallel counting and every execution
mode.

See jruby#5147.
…all fails its arity check

Unlike a def, whose arity check is an instruction in the body that runs after
ReceiveMethodCoverageInstr has taken the counter, the lambda arity check for a
define_method method runs in Java before the body. A call that failed it left
the entry's counter pending on the thread, and the next run of that block body
(for example a plain call of the same proc) took and counted it.

ProcMethod#call now clears the pending counter once the call returns or raises,
on the path where the entry has a counter at all.

Addresses review feedback on jruby#9676
The initialize of a Ruby subclass of a Java class does not run through
DynamicMethod#call: ConcreteJavaProxy runs it up to its super call
through the exitable interpreter (or a compiled terminal split), and
when the body is a plain super forwarding its arguments it does not run
it at all. None of those paths handed the entry's coverage counter to
the body, so such an initialize reported zero calls.

The hand-off now happens in AbstractIRMethod#startSplitSuperCall, with
the compiled-or-interpreted choice split out into splitSuperCall so that
MixedModeIRMethod can still delegate to its jitted method (which is not
the registered entry and holds no counter). The two ConcreteJavaProxy
shortcuts that skip the Ruby body altogether count the call directly;
they have already established that the arguments match the arity, which
is where MRI fires the CALL event it counts.
…counted

The parser grew and filled its array of starting line counts for every
parse made while coverage was set up, including Coverage.start(methods:
true), where CoverageData#prepareCoverage never looks at it, and
oneshot_lines, where it starts from an empty list instead. The file is
still registered with Coverage in every mode so that it appears in the
result.
Registration, clearing and result conversion all run while holding the
CoverageData lock, so the copy-on-write list was copying the whole array
on every method definition for nothing.
@sferik

sferik commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

@kares Thank you for the thoughtful review. I believe I have addressed all of your initial feedback. Please let me know if there are any additional changes you’d like me to make.

@headius

headius commented Sep 14, 2026

Copy link
Copy Markdown
Member

Master is 10.1 and that old branch should just be deleted. You did the right thing! I will be in flight soon and working most of the time so I'll try to have a look at these changes then.

@headius

headius commented Sep 15, 2026

Copy link
Copy Markdown
Member

I have removed the 10.1-dev branch to avoid future confusion.

@kares

kares commented Sep 15, 2026

Copy link
Copy Markdown
Member

I believe I have addressed all of your initial feedback. Please let me know if there are any additional changes you’d like me to make.

Took a look, some of the changed are a bit invasive but don't really have an alternative moving forward atm.
Thing worth investigating is how MRI implements the coverage approach, they seem to have it more centralized.

If it's okay with @headius to ship like this and potentially do more refactoring later I think it's good enough...
esp. if you're Java/JVM experience is low, green field features are IMO a good starting point - great job. 💟


p.s. Had a bit of a struggle reading some of the technical comment text (seems like my comment about that got lost with GH), try to instruct AI to compact/clarify further, maybe to use simplified technical English.

@headius

headius commented Sep 15, 2026

Copy link
Copy Markdown
Member

I'll give it a look once I'm settled at my hotel, tonight or tomorrow. Thanks again for the effort @sferik!

… class

MRI counts a method call from the method entry stored in the VM frame,
so no call path there needs to know about coverage. JRuby frames carry
only the implementation class and name, so the entry's own call methods
are the only place that can pass its counter to the body.

Each arity overload of InterpretedIRMethod and MixedModeIRMethod already
repeated the same preamble (debug output, JIT promotion). Fold that and
the coverage hand-off into a single prepareCall per class, so the
hand-off is invoked from one place instead of once per overload.
Shorter sentences, no asides in parentheses, and one idea per sentence,
so the comments are easier to read for non-native speakers.

Also move the pending coverage field in ThreadContext above the
constructor's javadoc, which it had been inserted under by mistake, and
regenerate the parser for the changed grammar comment.
@sferik

sferik commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

@kares Thanks again for your review.

I compared this against MRI (ext/coverage, thread.c, vm_method.c). MRI is centralized because its VM stores the method entry in every frame: the CALL hook reads cfp->me and bumps a hash counter, so no call path knows about coverage. JRuby frames carry only the implementation class and name, not the entry, so the hand-off has to start in the entry’s own call methods. Everything else already mirrors MRI: registration on putMethod matches method_added recording into me_set, forwarding entries resolve to the original, and results are built per file from the same keys.

I did reduce the number of call sites: InterpretedIRMethod and MixedModeIRMethod now do the hand-off from a single prepareCall, which also holds the debug/JIT preamble those overloads already repeated, instead of one line per arity overload. CompiledIRMethod keeps its per-overload line because each invokeExact has its own exact signature. I also rewrote all the comments this PR adds in shorter, plainer English.

An eval that is not covered was parsed with coverage off entirely, so
its methods were built without the counting instructions. The entry was
still registered, so such a method appeared in the result with a count
stuck at zero. MRI leaves only the lines of an uncovered eval out of the
result; the calls of the methods it defines are counted like any other.

The parse of an eval now emits the method coverage instructions whenever
methods mode is on, independent of the eval option, which only decides
whether lines are counted. Registration also skips a method whose scope
was parsed without those instructions, which is still the case for code
parsed before methods mode was on, so such a method is left out of the
result instead of reporting a count that can never move.
A block passed to a call whose arguments are not parenthesized, and a
block passed to a call of a method named like a constant, come through
grammar rules of their own: cmd_brace_block, do_block, and the
primary_value tCOLON2 tCONSTANT rule. None of them recorded where the
block starts and ends, so method coverage reported -1 for both columns
of a method defined from such a block.

Worse, none of them set the block's line from its opening brace or do
keyword either, so the line came from the empty production that starts
the block body, whose start position is left over from whatever was
parsed before it. A call spanning several lines therefore reported the
line its arguments start on rather than the line of the block. That line
is what TracePoint reports for b_call too, so both are now correct.

The three rules do what brace_block already did, and the keys method
coverage reports for these blocks now match MRI.
CoverMethodInstr decodes an operand but wrote none, since neither
OneOperandInstr nor Instr writes operands, so only the operation code
went out. Persisting IR with -Xir.writing while methods mode was on
therefore produced a file that goes out of step at the first
COVER_METHOD and reads back as garbage.
@sferik

sferik commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor Author

I found a few more bugs and pushed some fixes (and tests) to make this implementation consistent with CRuby. Let me know if you want me to squash all of these down into one commit.

@headius I believe this should be in pretty good shape for you to review. Much appreciated!

IRRuntimeHelpers.coverLine promised "true if the line was counted" but
returned true as soon as coverage was running, before CoverageData
decided whether it had anywhere to put the count. CoverageData.coverLine
gives up silently on a negative line, an untracked file, a file with no
line counts, and a line past the end of them.

Both callers now act on that answer: the interpreter and the indy
coverage site disarm a oneshot probe once the line is counted, so a
probe whose line had nowhere to go disarmed itself and lost the line for
good. It now stays armed, which is what the caller already assumed.
IRScope.endLine is -1 when no source span was recorded, which is still
the case for methods built by Prism. Adding one to convert it to a
1-based line turned that marker into 0, a line number that looks real.
The key then carried two different markers at once, as in
[Foo, :bar, 3, -1, 0, -1].
define_method(:new_name, some_method) binds a RenamedDynamicMethod that
wraps a copy of some_method, and neither the wrapper nor the copy ever
learns the new name: only the id passed to putMethod knows it. The key
therefore carried the name the method was copied from, so

  class A; def foo; end; end
  B.send(:define_method, :bar, A.instance_method(:foo))

reported [B, :foo, ...] where MRI reports [B, :bar, ...]. MRI keys the
entry by its called_id, the name it was defined under, which is what
putMethod already has in hand.
A counter was attached to its DynamicMethod for good: resetCoverage only
dropped the map of results. Three things followed a single completed
run, for the rest of the process.

InvokeSite.buildJittedHandle refuses to bind a call site directly to the
compiled body of a counted method, since the hand-off happens in
DynamicMethod.call. With the counter still attached, every method
defined during the run stayed permanently unbindable, long after anyone
was measuring. Calls also kept writing the counter to the ThreadContext,
and each counter kept an IRScope and a RubyModule alive.

resetCoverage now detaches every counter and invalidates the owners
whose entries changed, so call sites bound to the counting path
re-resolve and can bind directly again. The invalidation runs outside
the CoverageData lock: it takes the hierarchy lock, while a thread
defining a method holds the method table lock and then waits in
registerMethod, so holding both at once could deadlock.

The counter holds its entry weakly, so a method discarded during a run
is not kept alive until the run ends.
ExitableInterpreterContext decides whether the initialize of a Java
subclass can be skipped and its super called directly by walking the
instructions ahead of the super and rejecting anything it does not
recognize. The two instructions method coverage adds sit in exactly that
range: ReceiveMethodCoverageInstr is the first instruction of the body
and CoverMethodInstr follows argument receipt.

So with methods mode on, every one of those scans failed for every
instrumented method, and

  class Sub < java.util.ArrayList
    def initialize(x)
      super(x)
    end
  end

fell back to the split interpreter on each Sub.new. The counts stayed
right, since the fallback hands off the counter, but DynamicMethod's
coverElidedCall and the two ConcreteJavaProxy shortcuts it serves became
unreachable whenever a counter was attached, which is the only time they
have anything to do.

Skipping the two instructions is safe: it loses only the call count, and
coverElidedCall records that on the paths that take the shortcut. The
test now asserts the shortcut is really taken, since its counts pass
either way.
cover() is a lock-free atomic add, but clear() was a plain store. A
counter is cleared under the CoverageData lock and counted without it,
so Coverage.result(clear: true) on a running program discarded the
increment of every call that was in flight, one per call.

The parallel counting test never caught this because it does not clear.
peek_result held the lock for the whole conversion so that another
thread could not parse a file or define a method while the result was
being built. The methods hash is keyed by an Array, and hashing one
dispatches Ruby's Array#hash, which dispatches Module#hash on the owner.
So arbitrary Ruby ran under the lock, and it can define a method, which
needs the method table lock.

RubyModule.addMethodInternal takes the two in the other order: it holds
the method table lock and then waits in registerMethod. A thread calling
Coverage.result and a thread defining a method could therefore deadlock.
resetCoverage already avoids exactly this, which is why it compares
modules by identity and invalidates them outside the lock.

The lock now only covers a snapshot of the per-file data. The conversion
runs after it is released. The snapshot shares the counters themselves,
since all that is read from them is final state and a volatile count.
RubyModule.putMethod calls registerMethod for every method entry
whenever coverage is set up, whatever its mode. registerMethod was
synchronized as a whole and decided it had nothing to do only after
taking the lock.

Under a plain Coverage.start, which is what SimpleCov uses, every def,
define_method, attr_accessor, alias and visibility change in the process
therefore contended for the one lock that coverLine takes on every
executed line, to do nothing.

The mode check and the forwarding-entry checks now run before the lock.
Coverage.result(clear: true) built the result and then cleared the
counts as two separate steps, each taking the CoverageData lock. A call
counted between them was reported nowhere: it was not in the result that
had already been built, and its count was then zeroed.

The copy a result is built from now does the clearing as it reads, so
every count is either in the result or still in the counter. For the
method counters that is one atomic read-and-reset, since they are
counted without the lock.

clearCoverage is gone, as is MethodCoverage.clear. Coverage.result was
their only caller.
@headius

headius commented Sep 16, 2026

Copy link
Copy Markdown
Member

This is a big piece of work I think should delay until 10.1.3.0 (I'm hoping to release 10.1.2.0 shortly and this will not have enough bake time). I'm marking it as such and will review for merge soon after the release.

@headius headius modified the milestones: JRuby 10.0.x, JRuby 10.1.3.0 Sep 16, 2026
@sferik

sferik commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

@headius Congrats on shipping 10.1.2.0 on Monday. Are you able to review this patch now? Could you also please take a look at #9682?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants