[Test][Zeta] Drive FileCollectReader idle close and rediscovery with a manual clock - #12472
Open
goutamadwant wants to merge 1 commit into
Open
goutamadwant wants to merge 1 commit into
goutamadwant wants to merge 1 commit into
Conversation
…a manual clock FileCollectReaderBehaviorTest.rediscoversFileAfterInactiveCursorClosed inferred the inactive close and the glob rediscovery from wall-clock arithmetic with 20-30 ms of slack. A single scheduler stall near the idle deadline moves the closing poll late enough that rediscovery happens only after "second" is appended. With read_from_beginning=false the rediscovered file is opened at its end, the line is skipped, and the final assertion times out with "expected: <1> but was: <0>" (seen on Windows JDK 8 and JDK 11 unit-test jobs). FileCollectReader and FileTailCursor now take an optional millisecond clock (LongSupplier). The existing constructors keep using System::currentTimeMillis, so runtime behaviour is unchanged. The test steps the clock past close_inactive_ms and then one glob scan interval, so each transition happens in a known poll. All original assertions on the emitted event are kept, and the test now also checks that line numbering restarted, which only happens if the cursor was really closed and the file rediscovered.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose of this pull request
FileCollectReaderBehaviorTest.rediscoversFileAfterInactiveCursorClosedfails intermittently on Windows unit-test jobs with:unit-test (8, windows-latest), [Fix][API] Correct nested row byte accounting #12305: https://github.com/goutamadwant/seatunnel/actions/runs/35673549024/job/106584161477unit-test (11, windows-latest), [Feature][Connector-V2] Add MariaDB JDBC and CDC connector support #12258: https://github.com/merlau/seatunnel/actions/runs/35990368552/job/107605215163Why it fails. The test infers two internal transitions from wall-clock arithmetic with 20–30 ms of slack: the inactive-cursor close (
close_inactive_ms = 500) and the glob rediscovery (glob_scan_interval_ms = 20). Inside onepoll(),FileCollectReaderruns discovery before it closes idle cursors, so rediscovery needs a later poll at least one scan interval after the closing poll.A single stall of a few tens of milliseconds near the idle deadline pushes the closing poll late. The second Awaitility loop then exits before the next scan is due, and
"second\n"is appended while the file is still untracked. Withread_from_beginning = falseand no stored position, the rediscovered file is opened at its end, so the new line is skipped. The finaluntilAssertedthen times out withexpected: <1> but was: <0>. A longer timeout cannot help, because the line is already behind the cursor.The change.
FileCollectReadergets a package-private constructor that takes a millisecond clock (LongSupplier). It uses that clock for the glob scan interval, idle-cursor bookkeeping and line timestamps, and passes it toFileTailCursor.FileTailCursorgets a matching constructor overload. It has to be public, becauseFileTailCursorlives in thecursorsubpackage.System::currentTimeMillis, so runtime behaviour does not change. TheupdatedAtvalue written to the position store still uses wall-clock time.close_inactive_msand then oneglob_scan_interval_ms, so each transition happens in a known poll. All the original assertions on the emitted event are kept: exactly one event, which containssecondand notfirst.linemetadata is1. Line numbering only restarts if the cursor was really closed and the file rediscovered, so the test no longer passes when the idle close does nothing.This overlaps #12381, which raises the same test's Awaitility budgets from 3 s to 10 s. In the failing runs, the time is lost before the append, not in the final wait, so the larger budget does not cover this case. If #12381 merges first, I will rebase.
Does this PR introduce any user-facing change?
No. The new constructors are for tests, and the default clock is
System::currentTimeMillis, as before.How was this patch tested?
Forced timing window on unchanged
dev(deb16a3c3). A local-only hook, not in this PR, inserted a single 45 msThread.sleepin the first Awaitility loop, 35 ms before the idle deadline. Each run executed the single method repeatedly in one JVM through the JUnit Platform launcher:dev, with hookOn
dev, every failure had the CI message (expected: <1> but was: <0>within 3 seconds). Without the hook,devpassed 50/50 per JDK on macOS under CPU contention. That fits a failure that needs a stall of about 20 ms or more at a specific moment, which Windows runners hit more often.For the patched run, the injected stalls were 45 ms before the close step, 700 ms after the close, and 700 ms after rediscovery. The 700 ms stalls are longer than
close_inactive_ms, which would re-close the cursor under a wall clock.Stress after the patch. 200/200 passed on JDK 8 and 200/200 on JDK 11, with 20 busy-loop processes competing for CPU.
Mutation checks. The patched test fails as expected in both cases:
globResolver.forget(...)removed fromcloseInactiveCursors, so the file is never rediscovered.closeInactiveCursors(now)removed frompoll(), so the cursor is never closed.Module builds.
./mvnw -B -o -pl seatunnel-edge-agent/seatunnel-edge-agent-connector,seatunnel-edge-agent/seatunnel-edge-agent-starter clean verify -DskipUT=false -DskipIT=truepassed on JDK 8 and JDK 11, including spotless: connector 38 tests and starter 103 tests, with 0 failures.Not run locally: Windows, and
seatunnel-edge-agent-e2e(EdgeAgentFileToEngineIT). The E2E suite needs the Docker engine setup; it runs in this PR'sedge-agent-itjob.Check list
New License Guide
incompatible-changes.mdto describe the incompatibility caused by this PR.