Skip to content

RATIS-2719. SegmentedRaftLog.close may deadlock with the cache eviction thread. - #1617

Merged
szetszwo merged 1 commit into
apache:masterfrom
slfan1989:RATIS-2719
Oct 1, 2026
Merged

szetszwo merged 1 commit into
apache:masterfrom
slfan1989:RATIS-2719

Conversation

@slfan1989

@slfan1989 slfan1989 commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

SegmentedRaftLog.close() acquires the write lock before calling cacheEviction.close(), which interrupts the cache eviction thread and waits for it to terminate using Thread.join().

If the eviction thread is waiting for the same write lock, a deadlock occurs:

image

the closing thread waits for the eviction thread to terminate, while the eviction thread waits for the closing thread to release the lock.

Interrupting the eviction thread does not resolve this because it acquires the lock using lock().

This PR moves cacheEviction.close() before acquiring the write lock, so the closing thread waits for the eviction thread to terminate without holding the lock. The eviction thread can therefore complete its current operation and exit before the cache is closed.

What is the link to the Apache JIRA

RATIS-2719. SegmentedRaftLog.close may deadlock with the cache eviction thread.

This issue was identified while investigating a timeout in TestSegmentedRaftLog.testAppendAndRoll during a CI run. The test timed out after 100 seconds, and the following stack trace shows that the test thread was waiting in Thread.join() while closing the cache eviction thread:

java.util.concurrent.TimeoutException:
testAppendAndRoll(java.lang.Boolean, java.lang.Boolean) timed out after 100 seconds

2026-09-29 10:18:07,404 [main] WARN util.AwaitToRun
- Thread[#237,s0@group-7F137C7CB308-cacheEviction-AwaitToRun,5,main]:
  join is interrupted

java.lang.InterruptedException
    at java.base/java.lang.Object.wait0(Native Method)
    at java.base/java.lang.Object.wait(Object.java:389)
    at java.base/java.lang.Thread.join(Thread.java:1887)
    at java.base/java.lang.Thread.join(Thread.java:1963)
    at org.apache.ratis.util.AwaitToRun.close(AwaitToRun.java:102)
    at org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.close(SegmentedRaftLog.java:536)
    at org.apache.ratis.server.raftlog.segmented.TestSegmentedRaftLog.testAppendAndRoll(TestSegmentedRaftLog.java:449)

https://github.com/apache/ratis/actions/runs/36554382990/job/109359915576?pr=1615

How was this patch tested?

https://github.com/slfan1989/ratis/actions/runs/36564811620

@amaliujia amaliujia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch!

@szetszwo szetszwo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 the change looks good.

@szetszwo
szetszwo merged commit 68d2167 into apache:master Oct 1, 2026
16 checks passed
@szetszwo

szetszwo commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

@slfan1989 , thanks for working on this!

@amaliujia , thanks also for reviewing this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants