Conversation
|
|
||
| @Override | ||
| public IRubyObject call(ThreadContext context, IRubyObject self, RubyModule clazz, String name, IRubyObject[] args, Block block) { | ||
| prepareMethodCoverage(context); |
There was a problem hiding this comment.
is this approach the best we can do, having to invoke prepareMethodCoverage from all places?
not sure how much thought went into the PR but would be nice to consider other options...
There was a problem hiding this comment.
I did weigh alternatives but open to suggestions I may not have considered.
The counter has to be per method entry (CRuby keys results by [owner, name, location], so Class.new { def x; end } in a loop or %w[a b].each { |n| define_method(n) { } } produce distinct counts from one scope), and the count has to happen after arguments are bound (CRuby counts on the CALL event, so a call that dies on arity or a missing keyword is not counted). So something has to get the entry's counter from the DynamicMethod into the body, and the body has to do the increment.
Here are the options I considered:
- Wrap the entry in a delegating
DynamicMethodat registration, the wayProfilingDynamicMethod/MethodEnhancerdo it, so the hand-off lives in one class. The problem is that a wrapper changes the concrete class of the entry and several places key real behavior off that:InvokeSitedecides keyword handling byentry.method instanceof AbstractIRMethod, refinementimport_methodsraises for non-AbstractIRMethodentries,Method#==fordefine_methodmethods checksinstanceof ProcMethod,ruby2_keywordsreads the static scope through the same check, and the Java proxy code detects user-definedinitialize/newwith it. Each would need an unwrap. That is more invasive and more fragile than one line per call variant, and it only matters while coverage is on. - I didn’t count in
callwithout a body probe because it’s the same number of touch points and wrong for calls that fail on their arguments. - Hook the trace
CALLevent won’t work because it’s only emitted in full-trace mode and before arguments are received. - I didn’t fold the hand-off into the compiled
MethodHandles because it coversCompiledIRMethodonly. Interpreted and mixed-mode paths still need explicit hooks. - I didn’t pass the entry into the body because it touches the interpreter, JIT, and indy signatures.
The one-liner per arity variant follows what these classes already do for every per-call concern: doDebug() and the callCount/tryJit check are repeated in each call of MixedModeIRMethod and InterpretedIRMethod. prepareMethodCoverage sits next to those. With coverage off it is a field load plus a predictable null check, which is cheaper than the callCount check beside it.
If you would rather see fewer sites, InterpretedIRMethod's ten variants could route through the five Interpreter.INTERPRET_METHOD helpers by passing the method instead of its scope, but that trades ten trivial lines for a signature change on a public helper. Happy to do that if you think it is worth it!
|
@sferik thank you for jumping in here and taking a shot at this! I agree with your analysis of how CRuby does this and I'm sure we can work through refining your patch to minimize performance impact. I'm going to mark this as 10.1.x so we can take a little more time to make it perfect. These missing coverage features have been bothering me more lately so I'm excited to get some help implementing them! |
|
@headius Thanks Charlie! I’m happy to help move these coverage features forward however I can, whether or not you actually use any of this code. I’ve added a bunch of features to SimpleCov in the past few releases, and I’d like for JRuby users to benefit from those changes. More selfishly, I’d like to remove the My experience with Java and the JVM is limited, so I’ll happily take whatever advice you have. I’m working through @kares feedback now. Targeting 10.1 sounds good to me. These features have been missing from JRuby for a while, so there’s no particular urgency and I think it’s more important that this is done right than done quickly. I’m particularly sensitive to any performance impact it might have on production code. If those impacts can’t be mitigated, I’m open to gating them behind |
|
@headius I just tried rebasing this branch from |
Coverage's methods mode now works and produces the same results as CRuby:
{ file => { methods: { [owner, name, start_line, start_column,
end_line, end_column] => call_count } } }
Design follows CRuby's per-method-entry counters and JaCoCo-style probes:
* A MethodCoverage counter is created for every method entry (each def
or define_method landing in a module, in RubyModule#putMethod) whose
source file is being tracked, and attached to the DynamicMethod.
Aliases, visibility changes of inherited methods and other forwarding
entries get no counter of their own. Their calls count toward the
original, as in CRuby.
* Every Ruby-level call path (interpreted, mixed-mode, JIT-compiled, and
proc-based define_method) hands the counter to the body through the
ThreadContext. The body takes it in its first instruction
(ReceiveMethodCoverageInstr) and counts the call once its arguments
have been received (CoverMethodInstr), so calls that fail on arity,
keywords or a raising default value are not counted, matching where
CRuby fires the CALL event. Blocks get the same probes since any block
may become a method via define_method. Indy call sites no longer bind
directly to the compiled handle of a method that is being counted.
* Counting is a lock-free atomic add, so threads calling the same method
in parallel neither serialize nor lose increments. Entries sharing a
key are summed like CRuby.
* The parser now records byte columns and end positions of def, block
and lambda definitions (DefNode/IterNode source spans), carried into
IRScope. Both generated parsers were regenerated from RubyParser.y
with jay.
Coverage data is now kept per file in FileCoverage (lines, methods, and
room for branches), results only contain the requested modes (branches
still reports an empty hash), files are listed in parse order, and
Coverage.supported?(:methods) is true.
Also fixes two pre-existing issues this work surfaced: IRClosure dropped
the coverageMode it was given, so blocks converted into methods for
define_method were built without any coverage and oneshot_lines probes
disarmed themselves when hit while coverage was suspended.
Un-excludes the CRuby method coverage tests and the :all-modes spec, and
adds JRuby-specific tests for parallel counting and every execution
mode.
See jruby#5147.
…all fails its arity check Unlike a def, whose arity check is an instruction in the body that runs after ReceiveMethodCoverageInstr has taken the counter, the lambda arity check for a define_method method runs in Java before the body. A call that failed it left the entry's counter pending on the thread, and the next run of that block body (for example a plain call of the same proc) took and counted it. ProcMethod#call now clears the pending counter once the call returns or raises, on the path where the entry has a counter at all. Addresses review feedback on jruby#9676
The initialize of a Ruby subclass of a Java class does not run through DynamicMethod#call: ConcreteJavaProxy runs it up to its super call through the exitable interpreter (or a compiled terminal split), and when the body is a plain super forwarding its arguments it does not run it at all. None of those paths handed the entry's coverage counter to the body, so such an initialize reported zero calls. The hand-off now happens in AbstractIRMethod#startSplitSuperCall, with the compiled-or-interpreted choice split out into splitSuperCall so that MixedModeIRMethod can still delegate to its jitted method (which is not the registered entry and holds no counter). The two ConcreteJavaProxy shortcuts that skip the Ruby body altogether count the call directly; they have already established that the arguments match the arity, which is where MRI fires the CALL event it counts.
…counted The parser grew and filled its array of starting line counts for every parse made while coverage was set up, including Coverage.start(methods: true), where CoverageData#prepareCoverage never looks at it, and oneshot_lines, where it starts from an empty list instead. The file is still registered with Coverage in every mode so that it appears in the result.
Registration, clearing and result conversion all run while holding the CoverageData lock, so the copy-on-write list was copying the whole array on every method definition for nothing.
c7ea845 to
032c5fc
Compare
|
@kares Thank you for the thoughtful review. I believe I have addressed all of your initial feedback. Please let me know if there are any additional changes you’d like me to make. |
|
Master is 10.1 and that old branch should just be deleted. You did the right thing! I will be in flight soon and working most of the time so I'll try to have a look at these changes then. |
|
I have removed the 10.1-dev branch to avoid future confusion. |
Took a look, some of the changed are a bit invasive but don't really have an alternative moving forward atm. If it's okay with @headius to ship like this and potentially do more refactoring later I think it's good enough... p.s. Had a bit of a struggle reading some of the technical comment text (seems like my comment about that got lost with GH), try to instruct AI to compact/clarify further, maybe to use simplified technical English. |
|
I'll give it a look once I'm settled at my hotel, tonight or tomorrow. Thanks again for the effort @sferik! |
… class MRI counts a method call from the method entry stored in the VM frame, so no call path there needs to know about coverage. JRuby frames carry only the implementation class and name, so the entry's own call methods are the only place that can pass its counter to the body. Each arity overload of InterpretedIRMethod and MixedModeIRMethod already repeated the same preamble (debug output, JIT promotion). Fold that and the coverage hand-off into a single prepareCall per class, so the hand-off is invoked from one place instead of once per overload.
Shorter sentences, no asides in parentheses, and one idea per sentence, so the comments are easier to read for non-native speakers. Also move the pending coverage field in ThreadContext above the constructor's javadoc, which it had been inserted under by mistake, and regenerate the parser for the changed grammar comment.
|
@kares Thanks again for your review. I compared this against MRI ( I did reduce the number of call sites: |
An eval that is not covered was parsed with coverage off entirely, so its methods were built without the counting instructions. The entry was still registered, so such a method appeared in the result with a count stuck at zero. MRI leaves only the lines of an uncovered eval out of the result; the calls of the methods it defines are counted like any other. The parse of an eval now emits the method coverage instructions whenever methods mode is on, independent of the eval option, which only decides whether lines are counted. Registration also skips a method whose scope was parsed without those instructions, which is still the case for code parsed before methods mode was on, so such a method is left out of the result instead of reporting a count that can never move.
A block passed to a call whose arguments are not parenthesized, and a block passed to a call of a method named like a constant, come through grammar rules of their own: cmd_brace_block, do_block, and the primary_value tCOLON2 tCONSTANT rule. None of them recorded where the block starts and ends, so method coverage reported -1 for both columns of a method defined from such a block. Worse, none of them set the block's line from its opening brace or do keyword either, so the line came from the empty production that starts the block body, whose start position is left over from whatever was parsed before it. A call spanning several lines therefore reported the line its arguments start on rather than the line of the block. That line is what TracePoint reports for b_call too, so both are now correct. The three rules do what brace_block already did, and the keys method coverage reports for these blocks now match MRI.
CoverMethodInstr decodes an operand but wrote none, since neither OneOperandInstr nor Instr writes operands, so only the operation code went out. Persisting IR with -Xir.writing while methods mode was on therefore produced a file that goes out of step at the first COVER_METHOD and reads back as garbage.
|
I found a few more bugs and pushed some fixes (and tests) to make this implementation consistent with CRuby. Let me know if you want me to squash all of these down into one commit. @headius I believe this should be in pretty good shape for you to review. Much appreciated! |
IRRuntimeHelpers.coverLine promised "true if the line was counted" but returned true as soon as coverage was running, before CoverageData decided whether it had anywhere to put the count. CoverageData.coverLine gives up silently on a negative line, an untracked file, a file with no line counts, and a line past the end of them. Both callers now act on that answer: the interpreter and the indy coverage site disarm a oneshot probe once the line is counted, so a probe whose line had nowhere to go disarmed itself and lost the line for good. It now stays armed, which is what the caller already assumed.
IRScope.endLine is -1 when no source span was recorded, which is still the case for methods built by Prism. Adding one to convert it to a 1-based line turned that marker into 0, a line number that looks real. The key then carried two different markers at once, as in [Foo, :bar, 3, -1, 0, -1].
define_method(:new_name, some_method) binds a RenamedDynamicMethod that wraps a copy of some_method, and neither the wrapper nor the copy ever learns the new name: only the id passed to putMethod knows it. The key therefore carried the name the method was copied from, so class A; def foo; end; end B.send(:define_method, :bar, A.instance_method(:foo)) reported [B, :foo, ...] where MRI reports [B, :bar, ...]. MRI keys the entry by its called_id, the name it was defined under, which is what putMethod already has in hand.
A counter was attached to its DynamicMethod for good: resetCoverage only dropped the map of results. Three things followed a single completed run, for the rest of the process. InvokeSite.buildJittedHandle refuses to bind a call site directly to the compiled body of a counted method, since the hand-off happens in DynamicMethod.call. With the counter still attached, every method defined during the run stayed permanently unbindable, long after anyone was measuring. Calls also kept writing the counter to the ThreadContext, and each counter kept an IRScope and a RubyModule alive. resetCoverage now detaches every counter and invalidates the owners whose entries changed, so call sites bound to the counting path re-resolve and can bind directly again. The invalidation runs outside the CoverageData lock: it takes the hierarchy lock, while a thread defining a method holds the method table lock and then waits in registerMethod, so holding both at once could deadlock. The counter holds its entry weakly, so a method discarded during a run is not kept alive until the run ends.
ExitableInterpreterContext decides whether the initialize of a Java
subclass can be skipped and its super called directly by walking the
instructions ahead of the super and rejecting anything it does not
recognize. The two instructions method coverage adds sit in exactly that
range: ReceiveMethodCoverageInstr is the first instruction of the body
and CoverMethodInstr follows argument receipt.
So with methods mode on, every one of those scans failed for every
instrumented method, and
class Sub < java.util.ArrayList
def initialize(x)
super(x)
end
end
fell back to the split interpreter on each Sub.new. The counts stayed
right, since the fallback hands off the counter, but DynamicMethod's
coverElidedCall and the two ConcreteJavaProxy shortcuts it serves became
unreachable whenever a counter was attached, which is the only time they
have anything to do.
Skipping the two instructions is safe: it loses only the call count, and
coverElidedCall records that on the paths that take the shortcut. The
test now asserts the shortcut is really taken, since its counts pass
either way.
2de8b59 to
b9ee86c
Compare
cover() is a lock-free atomic add, but clear() was a plain store. A counter is cleared under the CoverageData lock and counted without it, so Coverage.result(clear: true) on a running program discarded the increment of every call that was in flight, one per call. The parallel counting test never caught this because it does not clear.
peek_result held the lock for the whole conversion so that another thread could not parse a file or define a method while the result was being built. The methods hash is keyed by an Array, and hashing one dispatches Ruby's Array#hash, which dispatches Module#hash on the owner. So arbitrary Ruby ran under the lock, and it can define a method, which needs the method table lock. RubyModule.addMethodInternal takes the two in the other order: it holds the method table lock and then waits in registerMethod. A thread calling Coverage.result and a thread defining a method could therefore deadlock. resetCoverage already avoids exactly this, which is why it compares modules by identity and invalidates them outside the lock. The lock now only covers a snapshot of the per-file data. The conversion runs after it is released. The snapshot shares the counters themselves, since all that is read from them is final state and a volatile count.
RubyModule.putMethod calls registerMethod for every method entry whenever coverage is set up, whatever its mode. registerMethod was synchronized as a whole and decided it had nothing to do only after taking the lock. Under a plain Coverage.start, which is what SimpleCov uses, every def, define_method, attr_accessor, alias and visibility change in the process therefore contended for the one lock that coverLine takes on every executed line, to do nothing. The mode check and the forwarding-entry checks now run before the lock.
Coverage.result(clear: true) built the result and then cleared the counts as two separate steps, each taking the CoverageData lock. A call counted between them was reported nowhere: it was not in the result that had already been built, and its count was then zeroed. The copy a result is built from now does the clearing as it reads, so every count is either in the result or still in the counter. For the method counters that is one atomic read-and-reset, since they are counted without the lock. clearCoverage is gone, as is MethodCoverage.clear. Coverage.result was their only caller.
|
This is a big piece of work I think should delay until 10.1.3.0 (I'm hoping to release 10.1.2.0 shortly and this will not have enough bake time). I'm marking it as such and will review for merge soon after the release. |
|
@headius Congrats on shipping 10.1.2.0 on Monday. Are you able to review this patch now? Could you also please take a look at #9682? |
This adds the
methodsmode of theCoveragelibrary. Results have the same shape, keys and counts as CRuby:Keys are
[owner, name, start_line, start_column, end_line, end_column], exactly as CRuby reports them (1-based lines, 0-based byte columns,defthroughend, or the block/lambda passed todefine_method).Addresses the method half of #5147. Once this is merged, I intend to work on branch coverage, which I think would be substantially trickier.
MethodCoverageis the JRuby analogue of the per-method-entry counters CRuby keeps (itsme2counterhash). It is created inRubyModule#putMethod, the one place every method entry lands, whenever methods are being measured and the entry's source file is being tracked, and it is attached to theDynamicMethod. As in CRuby, entries that only forward to another entry get no counter of their own: aliases,private :inherited_methodin a subclass,define_method(:x, instance_method(:aliased)). Calls through them count toward the original.module_function,Class#dupanddefine_method(name, method_object)create fresh entries with the new owner, again as in CRuby.CRuby counts on the
CALLevent, which fires after the arguments have been bound, so a call that fails with anArgumentError(wrong arity, missing keyword) or a raising default value is not counted. To match that, the count happens inside the body rather than inDynamicMethod#call:InterpretedIRMethod,MixedModeIRMethod,CompiledIRMethod,ProcMethod) hands the entry's counter to the body through a slot onThreadContextReceiveMethodCoverageInstr, the first instruction of the body, takes the counter into a temp before any argument is received, so nested calls made while receiving arguments (default value expressions,to_aryconversions) cannot take itCoverMethodInstr, emitted right after argument receipt, increments it. The counter records which scope it belongs to, so a stale hand-off can never be attributed to another method.define_methodmethod whose lambda arity check fails (that check runs in Java, before the body) clears the counter it left pending, so a later plain call of the same proc is not counted as a call of the methodinitializeof a Java subclass, whichConcreteJavaProxyruns through the split-constructor machinery rather thanDynamicMethod#call(or skips entirely when it is a plain forwardingsuper), gets the hand-off instartSplitSuperCalland is counted directly when its body is skippedBlocks get the same two probes because any block can become a method via
define_method. When called as plain blocks they see no counter and the probes are a null check. Invokedynamic call sites no longer bind directly to the compiled handle of a method that is being counted, so the hand-off is never bypassed.Counting is a single lock-free atomic add (
VarHandle#getAndAdd), so threads calling the same method neither serialize nor lose increments. A new test verifies 8 threads × 5000 calls produce exactly 40000. Registration and result conversion are synchronized onCoverageDataas the existing line coverage code already is.The grammar now records byte columns and end positions for
def, blocks and lambdas (DefNode/IterNodegainedgetStartColumn/getEndColumn, andMethodDefNode#getEndLineis now the line of theendkeyword rather than the end of the body). Lambda spans follow CRuby: from the start of the parameter list, or just past->when there is none, through the end of the body. Both generated parsers were regenerated fromRubyParser.ywith jay; the toolchain was verified to reproduce the checked-in sources byte-for-byte before the grammar was touched. The spans are carried intoIRScopeand survive the block-to-method conversion used bydefine_method.CoverageDatanow keeps aFileCoverageper file (lines, methods, and room for branches) instead of a bareIntList. Results contain only the requested modes,branchesreports{}for now, files are listed in parse order like CRuby, andCoverage.supported?(:methods)is true.Two other notes:
IRClosuresilently dropped thecoverageModepassed to its constructor, so blocks converted into methods fordefine_methodwere built without any coverage instrumentation.oneshot_linesprobes disarmed themselves when hit while coverage was suspended, losing the line for good. They now stay armed until the line is actually counted.Known remaining differences, both pre-existing and unrelated to counting: JRuby keys loaded files by their real path, and
Class#dupkeeps aliases as aliases (CRuby materializes them into real entries). Prism builds do not supply columns yet, so keys would carry-1columns there.Following @enebo's note in #5147 (comment), branches will be marked at IR build time where the builder knows it is compiling a Ruby-semantic branch, rather than recovered from the IR's own jumps. The per-file container, the mode-driven result conversion and the parser source spans this PR adds are the groundwork for that.