Skip to content

RATIS-2781. Use daemon threads for client event loop groups - #1664

Open
yandrey321 wants to merge 2 commits into
apache:masterfrom
yandrey321:RATIS-2781
Open

yandrey321 wants to merge 2 commits into
apache:masterfrom
yandrey321:RATIS-2781

Conversation

@yandrey321

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Problem. The Netty DataStream client's worker group (NettyClientStreamRpc-workerGroup) runs non-daemon threads.
A JVM that has streamed data therefore doesn't exit when its main thread returns, unless it closes every client first.

  • NettyUtils.newEventLoopGroup uses ConcurrentUtils.newThreadFactory, which creates plain new Thread(runnable)
    threads. Each thread inherits the daemon flag of the thread that creates it.
  • Netty creates an event-loop thread lazily, from the thread that first submits a task to it, such as a connect. For a
    client, that is the application thread that opens the stream, often main.
  • The worker group is shared per JVM by default and shut down only when the last NettyClientStreamRpc is closed.
    Applications that rely on a shutdown hook to close their clients never get there: the hook can't run while a
    non-daemon thread is alive.

Apache Ozone hit this (HDDS-17278). Every MapReduce task that wrote with data streaming hung until the AppMaster
killed it after 60 s, which in TestDFSIO raised total job time from 15.2 to 27.8 minutes. Each stuck task's thread
dump shows a single non-daemon thread besides DestroyJavaVM: NettyClientStreamRpc-workerGroup--thread1, in
epollWait.

Change.

  • ConcurrentUtils.newThreadFactory(namePrefix, daemon) and NettyUtils.newEventLoopGroup(name, size, useEpoll, daemon): new overloads that can create daemon threads. The existing methods delegate with daemon = false and
    behave as before (the thread inherits the creator's flag).
  • The client-side groups use daemon threads:
    • NettyClientStreamRpc worker group (the DataStream client);
    • NettyRpcProxy.PeerMap group (the Netty RPC client);
    • GrpcClientRpc client workers, used when raft.grpc.client.worker-group.size > 0. With the default of 0, gRPC
      already uses its own daemon threads.
  • Server groups (GrpcServicesImpl, NettyRpcService, NettyServerStreamRpc) are unchanged.

Closing a client works as before: the shared group is still released and shut down gracefully when the last client
is closed. The daemon flag only stops an unclosed client from keeping its JVM alive.

Generated-by: Claude Code (Claude Opus 5.5)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-2781

How was this patch tested?

  • New TestDataStreamClientDaemonThreads:
    • testDaemonEventLoopGroup: from a non-daemon thread, a group created with daemon = true runs tasks on daemon
      threads, while the existing newEventLoopGroup(name, size, useEpoll) still inherits the caller's flag.
    • testDataStreamClientThreadsAreDaemon: on a gRPC cluster with Netty DataStream, a non-daemon thread opens a
      stream and writes. While the client is still open, every NettyClientStreamRpc-workerGroup thread is a daemon
      thread.
  • Mutation check: with the DataStream worker group back to non-daemon, the cluster test fails with
    NettyClientStreamRpc-workerGroup--thread3 ... is not a daemon thread.
  • Existing tests: 223 tests passed in the datastream and netty test packages, TestGrpcEventLoops,
    TestRaftWithGrpc, TestRaftAsyncWithGrpc and TestCustomGrpcServices, with one exception:
    TestRaftAsyncWithNetty#testBasicAppendEntriesAsyncKillLeader. That test is already marked @Flaky("RATIS-2498").
    Run alone with clean builds, it failed with the same "Failed to find m3" assertion on unchanged master (1 of 2
    runs) as on this branch, so the failure predates this change.

@amaliujia

amaliujia commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Server groups are changed actually:
1.NettyRpcService uses new NettyRpcProxy.PeerMap(id.toString(), server.getProperties())

super(server::getId, id -> new NettyRpcProxy.PeerMap(id.toString(), server.getProperties()));

2. NettyServerStreamRpc.newClient builds a DataStreamClient, which uses NettyClientStreamRpc, to forward streams to other peers. Those forwarding threads become daemon threads as well. NettyClientStreamRpc eventually will be created against this
static EventLoopGroup newWorkerGroup(RaftProperties properties) {

It does not seem to be harmful to turn those server groups into daemon threads. But as the PR description says server groups are not changed, just pointing out to verify the intention and impact.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants