Skip to content

Don't flush empty buffers - #347

Merged
vickenty merged 1 commit into
DataDog:masterfrom
mar-kolya:mar-kolya/no-empty-buffer-flush
Nov 20, 2026
Merged

vickenty merged 1 commit into
DataDog:masterfrom
mar-kolya:mar-kolya/no-empty-buffer-flush

Conversation

@mar-kolya

@mar-kolya mar-kolya commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

The remaining() < lowerBoundSize + 1 check in StatsDProcessor still reserves a byte for the separator the processor wrote between messages (48823d8), before messages ended with their own newline. As a result:

  • an exact fit flushes the current buffer first, even when it is empty, so the sender writes a zero-length packet;
  • a message that exactly fills the rest of a buffer starts a new one instead.

Separately, when a multibyte message overflows an empty buffer, that empty buffer is flushed before the retry fails the same way.

The fix drops the + 1. When an empty buffer overflows, it rethrows the BufferOverflowException without flushing the buffer. Handlers still see BufferOverflowException for multibyte messages that don't fit, whether or not an earlier message is buffered, as sends_gauge_without_truncation expects.

Behaviour change: a message that exactly fills the rest of a buffer now joins it, so the client sends slightly fewer packets and its packets_sent telemetry drops a little.

Testing: the flush decisions live in the processing loop, not in StatsDProcessor.Encoder, so StatsDProcessorTest drives a processor and reads its outbound queue. It fails on any empty packet. It covers an exact fit, an exact fit into the rest of a packet, and a multibyte overflow both with an empty buffer and after a buffered chunk; both overflow cases expect BufferOverflowException. On master the first three fail and the last passes; all four pass with this change. I ran the full suite on JDK 22 with --release 8 outside Maven. It has the same 31 Unix-socket failures as master, where jnr-ffi cannot load on macOS arm64. CardinalityOverrideTest's 1-second timeouts flaked occasionally on both master and this branch in interleaved runs.

🤖 Generated with Claude Code

@mar-kolya
mar-kolya marked this pull request as ready for review November 13, 2026 12:35
@mar-kolya
mar-kolya requested a review from a team as a code owner November 13, 2026 12:35
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-13T12:41:16.809655Z a1b26cf Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Pipelines

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: f18cb4f | Docs | Give us feedback!

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a1b26cf6d0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +112 to +113
throw new InvalidMessageException(
MESSAGE_TOO_LONG, builder.toString());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Update the existing multibyte-overflow test for the new exception

When the worker flushes the first valid gauge before badgauge is queued, this branch now reports InvalidMessageException, but NonBlockingStatsDClientTest.sends_gauge_without_truncation still unconditionally requires BufferOverflowException at line 345. If the first gauge remains buffered, the retry at line 117 still reports BufferOverflowException. Consequently, that existing test now passes or fails depending on worker scheduling; both outcomes are reproducible with its oversized payload. Update the existing assertion alongside this exception change, accounting for both buffer states, to avoid intermittent suite failures.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 72bec24. An empty buffer that overflows now rethrows the BufferOverflowException, still without flushing the buffer. Both buffer states report the same exception, so sends_gauge_without_truncation is deterministic again. StatsDProcessorTest covers both states.

🤖 Generated by Claude Opus 5.5 with Claude Code

The remaining() check still reserves a byte for the separator the
processor wrote between messages before messages carried their own
newline. An exact fit therefore flushed the current buffer first, even
when it was empty, and a message that exactly filled the rest of a
buffer went to a new one. When a multibyte message overflowed an empty
buffer, that empty buffer was flushed before the retry failed the same
way.

Drop the extra byte. When an empty buffer overflows, rethrow the
BufferOverflowException without flushing it, so handlers see the same
exception whether or not an earlier message is buffered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mar-kolya
mar-kolya force-pushed the mar-kolya/no-empty-buffer-flush branch from 72bec24 to f18cb4f Compare November 17, 2026 13:48
@vickenty
vickenty merged commit ef95e76 into DataDog:master Nov 20, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants