Skip to content

ORC-2228: Abort the writer instead of calling close() when addRowBatch fails - #2724

Open
cxzl25 wants to merge 1 commit into
apache:mainfrom
cxzl25:ORC-2228
Open

cxzl25 wants to merge 1 commit into
apache:mainfrom
cxzl25:ORC-2228

Conversation

@cxzl25

@cxzl25 cxzl25 commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

When addRowBatch fails, WriterImpl currently calls close() from the catch block. This PR replaces that call with a private abortOnFailure() that only marks the writer closed, removes it from the MemoryManager and closes the PhysicalWriter. It also makes addRowBatch fail fast with an IOException when the writer is already closed.

Why are the changes needed?

Two problems come from calling close() on the failure path.

  1. close() is virtual. Hive's org.apache.hadoop.hive.ql.io.orc.WriterImpl overrides it as flushInternalBatch(); super.close();, and flushInternalBatch() hands the buffered batch back to addRowBatch. When a batch fails (for example a dictionary chunkIndex overflow on a large string column), the sequence isaddRowBatch -> close() -> flushInternalBatch -> addRowBatch -> ... with the same failing batch at every level, until StackOverflowError. Every level re-inserts the batch into the dictionary, so a single failure turns into hours of CPU before the task reports anything. We captured a live Spark task thread with 26,734 frames, 8,811 of them this loop. After the stack unwinds, Spark's abort() calls close() again and the whole recursion starts over.
  2. Even without a subclass, close() flushes the current stripe and writes a footer for a writer that just failed mid-batch. The result is a file that reads fine but silently misses rows. A failed write should leave a file that is obviously incomplete, not one that masquerades as complete.

This mirrors what Parquet did in PARQUET-2437 : record the failure, stop flushing, and let close() return without a footer.

How was this patch tested?

  1. Add UT

Was this patch authored or co-authored using generative AI tooling?

Yes, co-authored with Claude Code (Claude Fable 5.1)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant