Repository navigation
Accept messages that exactly fill the send buffer - #346
Conversation
The UTF-8 encoder rewrite in 3ec7da7 (4.4.4) rejects a message whose encoded size equals maxPacketSizeBytes, although it fits an empty send buffer. The retry into a fresh buffer fails the same way, so the message is dropped and BufferOverflowException goes to the error handler. For a multi-valued message, every remaining chunk is dropped with it, because the chunker fills packets up to exactly maxPacketSizeBytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
| } | ||
|
|
||
| @Test(timeout = 5000L) | ||
| public void sends_multivalued_distribution_chunk_that_exactly_fills_the_packet() |
There was a problem hiding this comment.
Would it make sense to test this directly with a StatsDProcessor unit test instead?
There was a problem hiding this comment.
Good call, done in 5e8e788. The client-level tests are gone; StatsDProcessorTest reads packets from the outbound queue instead. It covers an exact fit, a multibyte exact fit (bytes reach the limit, chars don't) and a two-chunk message whose first chunk is exactly full. All three fail on master with BufferOverflowException.
Side note: an exact fit still flushes an empty buffer first, because of the older remaining() < lowerBoundSize + 1 check. Happy to tighten that separately.
🤖 Generated by Claude Opus 5.5 with Claude Code
Replace the client-level tests with StatsDProcessorTest, which reads the encoded packets from the outbound queue instead of a UDP server. It also covers a multibyte message whose encoded size, not its char count, exactly fills the packet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| return head + new String(filler) + "\n"; | ||
| } | ||
|
|
||
| private List<String> process(Message message, int expectedPackets) throws Exception { |
There was a problem hiding this comment.
These tests need to touch every aspect of the processor just to test one condition. Would it be possible to test just the fixed method directly? We can isolate it in a new class (for example StatsDProcessor.Encoder) to make testing it easier.
There was a problem hiding this comment.
Done in 4c35740. The UTF-8 loop now lives in StatsDProcessor.Encoder, unchanged apart from the fix, and StatsDProcessorEncoderTest calls it directly with a plain ByteBuffer. It covers exact fits made of 1-, 2-, 3- and 4-byte chars, which all fail with >=, plus three overflow cases. StatsDProcessorTest is gone.
🤖 Generated by Claude Opus 5.5 with Claude Code
Move the encoding loop from ProcessingTask into StatsDProcessor.Encoder, unchanged apart from the fix, and replace the processor-level StatsDProcessorTest with StatsDProcessorEncoderTest, which calls the encoder with a plain ByteBuffer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| static final class Encoder { | ||
| private final int maxPacketSizeBytes; | ||
| private char[] charBuffer; | ||
| // + 4 so that we can check for buffer overflow without computing encoded length first |
There was a problem hiding this comment.
Would it make sense to move this comment to where we add 4?
There was a problem hiding this comment.
Yes, moved it to the constructor line in 4c7094f.
🤖 Generated by Claude Opus 5.5 with Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since 4.4.4 (3ec7da7, "Optimize UTF-8 encoding"),
writeBuilderToSendBufferthrowsBufferOverflowExceptionwhen a message's encoded size equalsmaxPacketSizeBytes, although it fits an empty send buffer. The retry into a fresh buffer fails the same check, so the message is dropped and the exception goes to the error handler. Before 4.4.4 theCharsetEncoderwrote straight into the send buffer and accepted an exact fit.Multi-valued messages (
DirectStatsDClient.recordDistributionValues) hit this regularly:writeValuesTopacks values while the chunk is<= maxPacketSizeBytes, so a chunk that lands exactly on the limit is dropped together with every later chunk of the same call. Client telemetry still counts the metric as sent, because it counts at enqueue.The fix changes
>=to>.byteBufferhas 4 bytes of slack, so the encoding loop stays in bounds. To test it directly, the encoding loop moves unchanged intoStatsDProcessor.Encoder, which each processing task owns.Testing:
StatsDProcessorEncoderTestcalls the encoder with a plainByteBuffer. Exact fits made of 1-, 2-, 3- and 4-byte characters fail withBufferOverflowExceptionundermaster's>=and pass with the fix. Three overflow cases throw and leave the buffer untouched: one byte over in ASCII, one byte over in multibyte, and longer than the space left in a partly filled buffer. I ran the full suite on JDK 22 with--release 8outside Maven, because the enforcer needs JDK 9–17. It has the same 31 failures before and after, all Unix-socket tests where jnr-ffi cannot load on macOS arm64. CI covers JDK 9–17.🤖 Generated with Claude Code