GH-3574: Preserve null counts when min/max exceed limit

[efegokdemir]

Summary

Preserve null_count when a column's min/max statistics exceed the metadata size limit.

The size guard is needed for min/max values, but null_count is a compact independent statistic and remains useful to readers. This keeps the null count available without reintroducing oversized min/max metadata.

Changes

• Emit null_count and nan_count before applying the min/max size limit for non-empty statistics.
• Keep min/max omission behavior unchanged for oversized statistics.
• Extend the binary statistics round-trip test to cover the preserved counts.

Testing

• git diff --check passed.
• The focused Maven test and Spotless check could not be run in this environment because no Java runtime is installed.

Compatibility note

For oversized, non-empty statistics, this changes the result from no statistics to counts without min/max. This is narrower than the unconditional null-count change reverted in #3688, but it still changes the previous all-or-nothing behavior. That compatibility question is under discussion.

Fixes #3574

[divjotarora]

Thanks @efegokdemir!

[divjotarora]

@efegokdemir My previous comment was not addressed, the current code still sets NaN count stats after the withinLimit check. I suggest we set null + NaN stats unconditionally and guard only min/max by the withinLimit check.

Proposed code:

formatStats.setNull_count(stats.getNumNulls());
if (stats.isNanCountSet()) {
    formatStats.setNan_count(stats.getNanCount());
}

if (!withinLimit(stats, truncateLength)) {
    return formatStats;
}

[efegokdemir]

Moved nan_count alongside null_count before the withinLimit early return; min/max remain size-gated. Added testNanCountIsPreservedWhenMinMaxExceedLimit, which forces the floating-point stats size check over the limit and verifies null count, NaN count, and omitted min/max. The pushed remote HEAD is d015ab76dd7d2c9bf79f5795d971ee1c3a568f23, and I verified both changed files at that SHA. git diff --check passed. Focused Maven/Spotless validation could not run: the wrapper reports “Unable to locate a Java Runtime” in this environment.

[wgtmac]

IIUC, this PR will hit the same issue as #3688?

cc @Fokko

[efegokdemir]

Thanks for flagging #3688. At the current head, this change emits null_count and nan_count only for non-empty stats that fail the min/max size check; it still creates counts-only metadata where the old code emitted none. That is narrower than #3688’s unconditional behavior, but it does change the all-or-nothing contract. I have corrected the PR description to make that scope and compatibility question explicit. I will hold further code changes pending maintainer direction on whether this contract change is acceptable for #3574.

[divjotarora]

@wgtmac Do you have a link to some discussion on why we reverted the previous change? #3688 has very little context

[wgtmac]

@divjotarora Sorry I cannot find any concrete evidence right now. If I remember it correctly, that change may break Iceberg to consume Parquet statistics. Perhaps @kevinjqliu @Fokko could provide more detail.

[Fokko]

The issue is described on the devlist: https://lists.apache.org/thread/fx96wnmkbmx7pcz2h89t592ztc8bcrn1 I'm unable to open it ritght now.

It had to do with the semantic meaning of isEmpty of the statistics. I believe it was something that we expect everything to be set on Iceberg when isEmpty is false. Ideally, we want to check what we break with this change.

[divjotarora]

@Fokko Thanks for the link. Unfortunately, it doesn't load for me, just says "Could not load this thread". Seems like this is an issue on the Iceberg side? Is it a general spec issue in Iceberg or localized to a subset of implementations?

[Fokko]

The build has been unblocked for me after #3855 (thanks @wgtmac for the quick review). I've published the jar locally using mvn install -DskipTests and bumped Iceberg to 1.19.0-SNAPSHOT (make sure that you're on the latest main that includes apache/iceberg#18347).

I ran the tests and it passed all the tests on the Iceberg side:

BUILD SUCCESSFUL in 56m 29s
130 actionable tasks: 65 executed, 65 up-to-date

I would have to check what the difference is between this one, and the previous PR, but this seems to play nicer with Iceberg 👍