Repository navigation
Empty ListVector and LargeListVector can expose offset buffers with writerIndex greater than capacity #1194
Description
Describe the bug, including details regarding any error messages, version, and platform.
ListVector and LargeListVector can expose an invalid offset buffer state when valueCount == 0.
For an empty list vector, the logical offset buffer should still contain the leading offset entry:
ListVector:(valueCount + 1) * 4 == 4bytesLargeListVector:(valueCount + 1) * 8 == 8bytes
However, in the empty-vector path, the offset buffer can have:
readerIndex: 0
writerIndex: 4
capacity: 0
or the equivalent writerIndex: 8, capacity: 0 for LargeListVector.
This violates the normal buffer invariant:
0 <= readerIndex <= writerIndex <= capacity
Downstream consumers that unwrap or serialize the Arrow buffer through Netty can then fail with:
IndexOutOfBoundsException: readerIndex: 0, writerIndex: 4
(expected: 0 <= readerIndex <= writerIndex <= capacity(0))
The issue is that setReaderAndWriterIndex() sets the offset buffer writer index based on valueCount * OFFSET_WIDTH, which is 0 for empty vectors. But list vectors still require one offset slot even when there are no values.
The same issue applies to both:
org.apache.arrow.vector.complex.ListVectororg.apache.arrow.vector.complex.LargeListVector
Expected behavior
For valueCount == 0, the offset buffer should still have enough capacity and readable bytes for the leading zero offset:
(valueCount + 1) * OFFSET_WIDTH
So:
- empty
ListVectorshould expose at least 4 bytes for offset[0] - empty
LargeListVectorshould expose at least 8 bytes for offset[0]
The first offset value should be zero.
Actual behavior
An empty list vector can expose an offset buffer with a non-zero writer index but zero capacity, causing Netty buffer validation to fail when the buffer is unwrapped or consumed.
Suggested fix
Update ListVector.setReaderAndWriterIndex() and LargeListVector.setReaderAndWriterIndex() so the offset buffer writer index is based on:
(valueCount + 1) * OFFSET_WIDTHFor the valueCount == 0 case, ensure the offset buffer has enough capacity for the leading zero offset before setting the writer index.
Care should be taken not to shrink the vector's future offset allocation size when allocating this empty sentinel offset buffer.
Additional context
This was observed downstream in Dremio after upgrading Arrow Java. The failure occurred while sending a record batch containing an empty list vector, where the send path unwraps Arrow buffers through Netty.
The downstream error was:
SYSTEM ERROR: IndexOutOfBoundsException: readerIndex: 0, writerIndex: 4
(expected: 0 <= readerIndex <= writerIndex <= capacity(0))
This issue is distinct from #1125. That issue involves UnionListReader.setPosition on a post-IPC empty list. This issue is about the offset buffer exported by empty ListVector / LargeListVector instances having an invalid writer-index/capacity relationship.
Activity
prashanthbdremio commentedon Jun 26, 2026
cc: @Yicong-Huang
telemenar commentedon Jul 1, 2026
There is some nuance here because there are two things in tension:
valueCount == 0still has one offset entry, so the exported offset buffer should containoffset[0] == 0.ListVector/LargeListVectorlifecycle behavior: freshly constructed, cleared, or otherwise empty vectors may legitimately have no allocated buffers yet.Either way, the current
setReaderAndWriterIndex()behavior for avalueCount == 0ListVectoris guaranteed to produce an invalid buffer state if the vector is still in the same state returned byListVector.empty().There are also a few ways to get into that state somewhat unexpectedly. For example,
ListVector.TransferImpl.splitAndTransfer()callsListVector.clear()on the destination vector. When the split length is0, that can release the destination buffers and leave the destination withallocator.getEmpty()as its offset buffer.Given the existing clear/reset behavior and the surrounding class hierarchy, I suspect persistent early enforcement would be tricky to implement cleanly. It would mean preserving or recreating the one-entry offset buffer across construction,
clear(), and other empty-vector transitions, which may conflict with existing assumptions that an empty vector can hold no buffers.So boundary enforcement may be the more practical direction: allow the internal empty/unallocated state, but materialize the required zero offset at API boundaries where Arrow physical layout matters.
The design questions I’m not fully sure about are:
setReaderAndWriterIndex(), are there other boundaries that need the same protection?getFieldBuffers()to allocate the required one-entry offset buffer as part of preparing the vector for export/serialization?splitAndTransfer()also enforce this before returning a schema-visible destination vector, or should that be left entirely to export-time handling?apacheGH-1194: Preserve empty list offset buffers
apacheGH-1194: Preserve empty list offset buffers
fix(arrow): an empty list's offsets no longer reach Arrow as its shar…
fix(arrow): an empty list's offsets no longer reach Arrow as its shar…
jarohen commentedon Oct 20, 2026
Hey folks - we hit this in XTDB too, in a slightly worse form: when the empty list's offset buffer is allocator.getEmpty(), it's NettyAllocationManager.EMPTY_BUFFER, a single ArrowBuf shared across the whole JVM.
setReaderAndWriterIndexthen sets that shared buffer's writer index to 4, and every other vector holding the empty buffer sees it. In our case that's other threads' validity buffers mid-serialisation, which then fail withIndexOutOfBoundsException: index: 0, length: 4 (expected: range(0, 0))or go out with the wrong length. Under 8 concurrent Flight streams, roughly 1 in 7 batches came out corrupted; with one stream it never happens, because a later vector resets the index to 0 before anything serialises it.So whatever #1229 settles on for freshly constructed or cleared vectors, it'd be worth making sure the empty-vector path never sets a non-zero index on the allocator's empty buffer: allocate the 4-byte [0] instead, or leave the index alone when capacity is 0.
See xtdb/xtdb#6231 for the write-up and workaround on our side.
Cheers 🙂
James