Skip to content

HDDS-17436. XceiverServerDomainSocket retries a failed accept() at once and spins while the datanode is out of file descriptors - #11798

Draft
Gargi-jais11 wants to merge 1 commit into
apache:masterfrom
Gargi-jais11:HDDS-17436
Draft

Gargi-jais11 wants to merge 1 commit into
apache:masterfrom
Gargi-jais11:HDDS-17436

Conversation

@Gargi-jais11

@Gargi-jais11 Gargi-jais11 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

When accept() itself fails in XceiverServerDomainSocket.run, the loop logs a WARN and calls accept() again right away. For example, accept(2) fails with "Too many open files" when the datanode is out of file descriptors. On Linux, the queued connection keeps failing, so the accept thread spins on a CPU core and logs thousands of WARN stack traces per second until descriptors are free again.

Changes done:
This change adds a 1-second pause before retrying when accept() itself fails, which is when no peer has been created yet. Rejections of accepted connections, such as at the xceiver limit, and idle accept timeouts behave as before. Shutdown isn't delayed, because stop() interrupts the pause.

KEPT in draft to save CI resources, will mark it ready to review once approved

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-17436

How was this patch tested?

added testAcceptFailureIsRetriedAfterPause to TestXceiverServerDomainSocket. Without the fix it fails with about 14,600 accept() calls in 500 ms; with the fix it passes.
Without fix: (added few lines from test runs)

2026-10-27 11:02:22,680 [627313eb-83af-4e95-97d1-b2a5308e15c2-XceiverServerDomainSocket] WARN  server.XceiverServerDomainSocket (XceiverServerDomainSocket.java:run(231)) - Got an exception. Peer null
java.net.SocketException: accept(2) error: Too many open files
	at org.apache.hadoop.hdds.TestXceiverServerDomainSocket.lambda$testAcceptFailureIsRetriedAfterPause$1(TestXceiverServerDomainSocket.java:647)
	at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:42)
	at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:103)
	at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
	at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34)
	at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82)
	at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(MockMethodAdvice.java:134)
	at org.apache.hadoop.net.unix.DomainSocket.accept(DomainSocket.java:234)
	at org.apache.hadoop.ozone.container.common.transport.server.XceiverServerDomainSocket.run(XceiverServerDomainSocket.java:210)
	at java.base/java.lang.Thread.run(Thread.java:1583)
2026-10-27 11:02:22,680 [627313eb-83af-4e95-97d1-b2a5308e15c2-XceiverServerDomainSocket] WARN  server.XceiverServerDomainSocket (XceiverServerDomainSocket.java:run(231)) - Got an exception. Peer null
java.net.SocketException: accept(2) error: Too many open files
	at org.apache.hadoop.hdds.TestXceiverServerDomainSocket.lambda$testAcceptFailureIsRetriedAfterPause$1(TestXceiverServerDomainSocket.java:647)
	at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:42)
	at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:103)
	at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
	at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34)
	at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82)
	at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(MockMethodAdvice.java:134)
	at org.apache.hadoop.net.unix.DomainSocket.accept(DomainSocket.java:234)
	at org.apache.hadoop.ozone.container.common.transport.server.XceiverServerDomainSocket.run(XceiverServerDomainSocket.java:210)
	at java.base/java.lang.Thread.run(Thread.java:1583)
2026-10-27 11:02:22,680 [627313eb-83af-4e95-97d1-b2a5308e15c2-XceiverServerDomainSocket] WARN  server.XceiverServerDomainSocket (XceiverServerDomainSocket.java:run(231)) - Got an exception. Peer null
java.net.SocketException: accept(2) error: Too many open files
.
.
.
org.opentest4j.AssertionFailedError: accept() was called 19061 times ==> 
Expected :true
Actual   :false

After fix:

2026-10-27 10:51:26,895 [b03e6da0-e68c-47b9-ae32-6ce2bda71c99-XceiverServerDomainSocket] WARN  server.XceiverServerDomainSocket (XceiverServerDomainSocket.java:run(231)) - Got an exception. Peer null
java.net.SocketException: accept(2) error: Too many open files
	at org.apache.hadoop.hdds.TestXceiverServerDomainSocket.lambda$testAcceptFailureIsRetriedAfterPause$1(TestXceiverServerDomainSocket.java:647)
	at org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:42)
	at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:103)
	at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
	at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34)
	at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82)
	at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(MockMethodAdvice.java:134)
	at org.apache.hadoop.net.unix.DomainSocket.accept(DomainSocket.java:234)
	at org.apache.hadoop.ozone.container.common.transport.server.XceiverServerDomainSocket.run(XceiverServerDomainSocket.java:210)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant