@szetszwo @adoroszlai While upgrading the ratis-examples module, I discovered an issue where the workerGroup might not be properly shut down, causing unit tests to fail with the following exception.

2025-05-21 05:09:14,262 WARN  util.LeakDetector (LeakDetector.java:assertNoLeaks(175)) - 29/30) numLeaks == 1 > 0, will wait and retry ...
2025-05-21 05:09:15,265 WARN  util.ReferenceCountedLeakDetector (ReferenceCountedLeakDetector.java:logLeakMessage(168)) - LEAK: (class org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoopGroup, count=3, value=org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoopGroup@756cf158)

java.lang.IllegalStateException: #leaks = 1 > 0, #leaks == set.size = 1

	at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:77)
	at org.apache.ratis.util.LeakDetector$LeakTrackerSet.assertNoLeaks(LeakDetector.java:100)
	at org.apache.ratis.util.LeakDetector$LeakTrackerSet.getNumLeaks(LeakDetector.java:94)
	at org.apache.ratis.util.LeakDetector.assertNoLeaks(LeakDetector.java:178)
	at org.apache.ratis.server.impl.MiniRaftCluster.shutdown(MiniRaftCluster.java:892)
	at org.apache.ratis.grpc.MiniRaftClusterWithGrpc.shutdown(MiniRaftClusterWithGrpc.java:97)
	at org.apache.ratis.examples.filestore.FileStoreStreamingBaseTest.testFileStoreStreamSingleFile(FileStoreStreamingBaseTest.java:83)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

I will walk through the entire process of analyzing the issue, in hopes that it can offer some troubleshooting guidance to others facing similar problems.

• Step 1 I suspected the issue was due to the close method of NettyClientStreamRpc not properly releasing resources. To address this, I added logic to forcibly shut down the workerGroup. This approach worked as expected in early tests, but it later caused other unit tests to hang.

NettyClientStreamRpc.java#L234-L242

The modification logic is as follows

connection.close();
if (!connection.isTerminated()) {
  connection.workerGroup.shutdownGracefully();
}

• Step 2 I began investigating the reference counting logic involved in the creation of NettyClientStreamRpc.

This logic increments the reference count at line 97 with new WorkerGroupGetter(current.join().retain()), and releases it at line 103 with return previous.join().release() ? null : previous;.

I added trace logs to the retain and release methods in ReferenceCountedLeakDetector to monitor the reference count of org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoopGroup.

From the logs, I found that 30 object references were created, but only 15 were released. This indicates that some logic is not properly releasing the NioEventLoopGroup objects.

There are two main call chains involved:

• RaftClientImpl

RaftClientImpl  
  └── DataStreamClient  
      └── NettyDataStreamFactory  
          └── NettyClientStreamRpc  
              └── WorkerGroupGetter.newInstance  

• NettyServerStreamRpc

NettyServerStreamRpc  
  └── DataStreamClient  
      └── NettyDataStreamFactory  
          └── NettyClientStreamRpc  
              └── WorkerGroupGetter.newInstance  

By examining the code, we can see that NettyServerStreamRpc#close() properly releases resources. This suggests that the issue lies with RaftClientImpl, which does not release its resources. Since RaftClientImpl is directly instantiated by the client in the test, we need to explicitly call close() to ensure proper resource cleanup.

• testSingleFile

• testMultipleFiles
  

With the modification, the unit tests now run and pass successfully.

@szetszwo @adoroszlai RATIS-2303 has passed all checks. Although some unit tests are failing, the fixes are relatively straightforward. I'm considering including them directly in this PR. Could you please review this pr? Thank you very much!