Skip to content

ORC-2226: Allow seeking to the end of stream in InStream.seek - #2722

Open
jeanlyn wants to merge 1 commit into
apache:mainfrom
jeanlyn:fix-ORC-2226-seek-eof
Open

jeanlyn wants to merge 1 commit into
apache:mainfrom
jeanlyn:fix-ORC-2226-seek-eof

Conversation

@jeanlyn

@jeanlyn jeanlyn commented Oct 22, 2026 •

Copy link
Copy Markdown

What changes were proposed in this pull request?

Scope the fix to exactly the failing condition:

  1. InStream.UncompressedStream.seek(long): accept desired == length only when no bytes were read for the stream (bytes == null) — i.e. the read planner assigned a zero-byte range to it — and park at EOF. All other seeks keep the existing behavior, so genuinely out-of-bounds seeks on real data still throw.
  2. InStream.CompressedStream.seek(long): same condition. It also resets currentCompressedStart, because the buffered chunk is dropped and the cached header position is no longer valid (otherwise a later seek back to that cached position would silently stay at EOF).
  3. InStream.UncompressedStream.toString(): null-guard currentRange (pre-existing NPE for streams without backing bytes, which would otherwise mask the real exception message).

Why are the changes needed?

The row group index of an all-null trailing row group legitimately points at the end of the stream (both the Java and C++ writers record such positions). When predicate pushdown (SARG) selects only that row group, StripePlanner.processStream computes a zero-byte read range for the stream (start == end), so the stream is created with bytes == null. UncompressedStream.seek() then only special-cases desired == 0 && bytes == null, and a seek to desired == length falls through the range loop and throws:

Caused by: java.lang.IllegalArgumentException: Seek in column 2 kind DATA to 8192 is outside of the data
    at org.apache.orc.impl.InStream$UncompressedStream.seek(InStream.java:...)

A full scan (no SARG) reads the same file fine, as does the C++ reader. See ORC-2226 for a minimal, self-contained reproduction (also reproduced on 2.1.2 and 2.3.1).

The condition is deliberately narrowed to bytes == null so that no behavior changes for streams that actually have data: for bytes != null, seeking to the stream end keeps the pre-existing semantics (accepted only at the end of the last disk range, rejected otherwise), and corrupt/truncated files still fail loudly instead of silently reading EOF.

How was this patch tested?

New unit tests:

  • TestInStream#testUncompressedSeekToEndOfStream — an UncompressedStream with no backing bytes: seek(length) is accepted, read() then returns -1, available() returns 0, and seeking past the end (length + 1) still throws.
  • TestInStream#testCompressedSeekToEndOfStream — same for CompressedStream.
  • TestInStream#testCompressedSeekBackFromEndOfStream — read, seek to the end, then seek back to the start: the cached chunk-header state must stay consistent and the re-read must return the correct value.
  • TestRecordReaderImpl#testSargOnAllNullTrailingRowGroup — end-to-end regression: writes an uncompressed file whose trailing row group is all-NULL for a string column, then reads it with a SARG that selects only that row group; asserts both the row count and that the values are all NULL (fails before the fix, passes after).
  • TestRecordReaderImpl#testSargOnAllNullMiddleRowGroup — boundary guard: an all-null row group in the middle of a stripe keeps working via the following row group's estimated end offset.

Full orc-core suite passes locally: 1532 tests, 0 failures, 0 errors.

Was this patch authored or co-authored using generative AI tooling?

Yes, co-authored using generative AI tooling. The analysis, fix and tests were reviewed and verified by a human before submission.

Generated-by: Kimi Code 2.1.1

Notes for reviewers

  • Seeks beyond the stream end (desired > length), and any non-zero seeks other than desired == length on streams with no backing bytes, still throw — genuinely corrupt files are not masked.
  • Behavior now matches the C++ reader, which tolerates seek-to-end.
  • Cross-module impact: none; no file-format or API changes. Subclasses (EncryptedStream, EncryptedCompressedStream) inherit the fix.

@jeanlyn
jeanlyn force-pushed the fix-ORC-2226-seek-eof branch from e63c171 to 810e161 Compare October 22, 2026 06:37
@cxzl25
cxzl25 requested a balanced review from Copilot October 22, 2026 07:28

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open issues
What changed in this PR

Fixes ORC-2226 by allowing InStream.seek(length) for streams that have no planned read ranges (zero bytes assigned), which is needed when predicate pushdown selects an all-null trailing row group whose index points at end-of-stream.

Changes:

  • Allow seek-to-EOF (desired == length) when bytes == null in both UncompressedStream and CompressedStream (and reset compressed cached state).
  • Null-guard UncompressedStream.toString() to avoid masking the real exception on zero-byte streams.
  • Add unit + regression tests covering seek-to-EOF behavior and SARG selection of all-null row groups.
File Description
java/​core/​src/​java/​org/​apache/​orc/​impl/​InStream.java Accept seek(length) for zero-byte planned streams; reset compressed cached state; guard toString() against currentRange == null.
java/​core/​src/​test/​org/​apache/​orc/​impl/​TestInStream.java Add focused unit tests for seek-to-EOF on uncompressed/compressed streams and seeking back after EOF.
java/​core/​src/​test/​org/​apache/​orc/​impl/​TestRecordReaderImpl.java Add end-to-end regression tests for SARG selection of all-null trailing/middle row groups.

Review effort: Lite

Comment thread java/core/src/test/org/apache/orc/impl/TestInStream.java Outdated
Comment thread java/core/src/test/org/apache/orc/impl/TestRecordReaderImpl.java Outdated
Comment thread java/core/src/test/org/apache/orc/impl/TestRecordReaderImpl.java Outdated
Comment thread java/core/src/test/org/apache/orc/impl/TestRecordReaderImpl.java Outdated
The row group index of an all-null trailing row group points exactly at
the end of the stream. When predicate pushdown selects only such a row
group, the read planner assigns zero bytes to the stream, and
UncompressedStream.seek() throws IllegalArgumentException because the
position is not covered by any disk range.

Accept seeking to the end of the stream as a valid EOF position in both
UncompressedStream.seek() and CompressedStream.seek(), scoped to streams
that have no backing bytes (bytes == null) so that genuinely
out-of-bounds seeks on real data still throw. CompressedStream also
resets the cached chunk header position since the buffer is dropped.
Also null-guard UncompressedStream.toString() for streams without
backing bytes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants