Repository navigation
RATIS-2781. Use daemon threads for client event loop groups - #1664
Open
yandrey321 wants to merge 2 commits into
Open
yandrey321 wants to merge 2 commits into
yandrey321 wants to merge 2 commits into
Conversation
Generated-by: Claude Code (Claude Opus 5.5)
Contributor
|
Server groups are changed actually: 2. NettyServerStreamRpc.newClient builds a DataStreamClient, which uses NettyClientStreamRpc, to forward streams to other peers. Those forwarding threads become daemon threads as well. NettyClientStreamRpc eventually will be created against this It does not seem to be harmful to turn those server groups into daemon threads. But as the PR description says server groups are not changed, just pointing out to verify the intention and impact. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Problem. The Netty DataStream client's worker group (
NettyClientStreamRpc-workerGroup) runs non-daemon threads.A JVM that has streamed data therefore doesn't exit when its main thread returns, unless it closes every client first.
NettyUtils.newEventLoopGroupusesConcurrentUtils.newThreadFactory, which creates plainnew Thread(runnable)threads. Each thread inherits the daemon flag of the thread that creates it.
client, that is the application thread that opens the stream, often
main.NettyClientStreamRpcis closed.Applications that rely on a shutdown hook to close their clients never get there: the hook can't run while a
non-daemon thread is alive.
Apache Ozone hit this (HDDS-17278). Every MapReduce task that wrote with data streaming hung until the AppMaster
killed it after 60 s, which in TestDFSIO raised total job time from 15.2 to 27.8 minutes. Each stuck task's thread
dump shows a single non-daemon thread besides
DestroyJavaVM:NettyClientStreamRpc-workerGroup--thread1, inepollWait.Change.
ConcurrentUtils.newThreadFactory(namePrefix, daemon)andNettyUtils.newEventLoopGroup(name, size, useEpoll, daemon): new overloads that can create daemon threads. The existing methods delegate withdaemon = falseandbehave as before (the thread inherits the creator's flag).
NettyClientStreamRpcworker group (the DataStream client);NettyRpcProxy.PeerMapgroup (the Netty RPC client);GrpcClientRpcclient workers, used whenraft.grpc.client.worker-group.size > 0. With the default of 0, gRPCalready uses its own daemon threads.
GrpcServicesImpl,NettyRpcService,NettyServerStreamRpc) are unchanged.Closing a client works as before: the shared group is still released and shut down gracefully when the last client
is closed. The daemon flag only stops an unclosed client from keeping its JVM alive.
Generated-by: Claude Code (Claude Opus 5.5)
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/RATIS-2781
How was this patch tested?
TestDataStreamClientDaemonThreads:testDaemonEventLoopGroup: from a non-daemon thread, a group created withdaemon = trueruns tasks on daemonthreads, while the existing
newEventLoopGroup(name, size, useEpoll)still inherits the caller's flag.testDataStreamClientThreadsAreDaemon: on a gRPC cluster with Netty DataStream, a non-daemon thread opens astream and writes. While the client is still open, every
NettyClientStreamRpc-workerGroupthread is a daemonthread.
NettyClientStreamRpc-workerGroup--thread3 ... is not a daemon thread.TestGrpcEventLoops,TestRaftWithGrpc,TestRaftAsyncWithGrpcandTestCustomGrpcServices, with one exception:TestRaftAsyncWithNetty#testBasicAppendEntriesAsyncKillLeader. That test is already marked@Flaky("RATIS-2498").Run alone with clean builds, it failed with the same "Failed to find m3" assertion on unchanged master (1 of 2
runs) as on this branch, so the failure predates this change.