Repository navigation
test: pin list layouts nested in structs and check that leaves inside lists read alone - #89
Conversation
… lists read alone - Golden files for three more list layouts: a nullable struct holding lists of nullable structs four lists deep, nullable lists of nullable structs, and lists written with empty chunks before, between and after populated ones. Projections read leaves one to four lists deep from them. - A Spark test reads leaves one to four lists deep, under structs, through both read paths, and checks each reads under a fifth of the bytes of the list around it, and matches Parquet with pruning on and off. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
ChristopherWirt
left a comment
ChristopherWirt
left a comment
There was a problem hiding this comment.
Reviewed the changes against #88 and found no blocking issues.
The three additional golden files extend the existing read and writer-layout checks without replacing existing fixtures. The new cases cover nullable struct elements, four nested list levels below nullable structs, and empty chunks around populated input chunks. Projection scans are materialized and compared with the in-memory arrays.
The Spark tests force decoding with a hash aggregate, count Hadoop reads through the existing counting filesystem, compare each leaf against its surrounding list, and check results against Parquet through both V1 and V2 with pruning enabled and disabled. The wide sibling fields make the I/O assertions meaningful.
The current Linux musl CI check passed. I did not rerun tests locally; the additional targeted Rust/Spark runs and mutation check are reported in the PR description.
Summary
Closes #88. The list layout golden files pinned lists one or two deep. This adds the deeper shape wide event
schemas write, and a Spark test that leaves inside those lists are read without their siblings.
Changes
vortex-jni/goldenfiles/list_layout/:event_shape: a nullable struct holding nullable lists of nullable structs, which hold a nullable struct withits own list, a list of strings, and lists nested four deep. Every struct has a string sibling.
nullable_struct_elements: nullable lists of nullable structs.zero_row_chunks: nullable lists written as one stream of populated chunks with empty chunks before, betweenand after them.
event_shapeone, two and four lists deep, below a struct, and a list of strings.VortexListElementPruningIoTest: a table of that shape with eight wide string siblings per struct, written withand without
vortex.listLayoutand as Parquet. For leaves one to four lists deep it checks, through V1 and V2,that the list layout reads under a fifth of the bytes of the list around the leaf, and that results match Parquet
with nested schema pruning on and off. Filters on list fields are covered too.
Testing
cargo test -p vortex-jni --lib list_layout_golden: 77 passed (52 before).Writing
i64offsets fails the write check of all 10 cases, including the 3 new ones.VortexListElementPruningIoTeston Spark 4.0 and 3.5: 14 passed each. Bytes read, list layout, either path:cargo clippy -p vortex-jni --all-targets --all-features -- -D warnings,cargo +nightly fmt,spotlessApply.