Skip to content

test(jni): pin the vortex.list layout with golden files, and rename its edition to qc2026.11.0 - #85

Merged
ChristopherWirt merged 4 commits into
developfrom
cwirt/list-layout-golden-files
Nov 4, 2026
Merged

ChristopherWirt merged 4 commits into
developfrom
cwirt/list-layout-golden-files

Conversation

@ChristopherWirt

@ChristopherWirt ChristopherWirt commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Why

  • vortex.listLayout writes the vortex.list layout, in our draft list layout edition.
  • Upstream keeps vortex.list in its unstable edition, so it can change the format in any merge.
  • The event store writes its files this way, so a silent format change would leave files nothing can read.

What

Golden files in vortex-jni/goldenfiles/list_layout/, written once through the vortex.listLayout write path. They cover primitive, nullable, string, struct and nested list elements, a list inside a struct, and all-empty lists, each 20,000 rows across several blocks.

Every build checks two things:

  • Reads: each file still reads back to its values, through full scans, row ranges, row indices, projections, filters, list_contains and list_transform pushdown.
  • Writes: today's writer still writes the same list layouts: their metadata, children, dtypes and row counts.

The files are never regenerated by the check. A separate #[ignore]d test only writes missing files, for adding cases. The README explains what to do when a check fails.

Rename the edition to qc2026.11.0

  • It was jni2026.11.0, named after the vortex-jni crate. An upstream jni family in that crate would collide with it.
  • Edition names aren't stored in files, so files already written, including these golden files, read the same.
  • Eventstore records the name, so it must move to this build and qc2026.11.0 together.

Make qc2026.11.0 immutable, freezing in 0.91.0

  • Declared immutable: true, with min_library_version still None. Vortex freezes an edition by setting min_library_version to the release that freezes it.
  • Set it to "0.91.0" once the fork's first 0.91.0 build is cut. Changes landed after 0.90.0, so "0.90.0" would claim readers that may not read these files.
  • The golden files hold the format fixed from now on either way.
  • list_layout_edition_becomes_immutable fails if a merge makes it mutable.

Checked

I broke the format three ways, and the tests caught each one:

Change Failed
Writer stores i64 offsets 7 write checks
Reader swaps the elements and offsets children all 52
ListLayoutMetadata field renumbered 41

Testing

  • cargo test -p vortex-jni --lib: 71 passed.
  • cargo clippy -p vortex-jni --all-targets --all-features -- -D warnings: clean.
  • cargo +nightly-2026-09-10 fmt -p vortex-jni: applied.

🤖 Generated with Claude Code

ChristopherWirt and others added 2 commits November 4, 2026 11:29
The list layout is in the draft jni2026.11.0 edition, so nothing upstream
promises it stays readable. Add golden files written through the
vortex.listLayout write path, and check on every build that:

- each file still reads back to its values, through full scans, row
  ranges, row indices, projections, filters and list_transform pushdown;
- today's writer still writes the same list layouts: metadata, children,
  dtypes and row counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
The family was named after its crate, so an upstream jni family in
vortex-jni would collide with it. Edition names aren't stored in files,
so files already written read the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
@ChristopherWirt ChristopherWirt changed the title test(jni): pin the vortex.list layout with golden files test(jni): pin the vortex.list layout with golden files, and rename its edition to qc2026.11.0 Nov 4, 2026
Declare it immutable with a min_library_version, so it promises its
files stay readable. The golden files enforce the promise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
@ChristopherWirt ChristopherWirt changed the title test(jni): pin the vortex.list layout with golden files, and rename its edition to qc2026.11.0 test(jni): pin the vortex.list layout with golden files, and freeze it as qc2026.11.0 Nov 4, 2026
Keep it immutable but unfrozen until the fork's first 0.91.0 build, so
min_library_version names a release whose readers all read it. The
golden files already hold the format fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
@ChristopherWirt ChristopherWirt changed the title test(jni): pin the vortex.list layout with golden files, and freeze it as qc2026.11.0 test(jni): pin the vortex.list layout with golden files, and rename its edition to qc2026.11.0 Nov 4, 2026

@gitRasheed gitRasheed left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, fyi hours written so far record jni2026.11.0, so its checkReadable will reject them on this build but don't think it matters

@ChristopherWirt
ChristopherWirt merged commit 36e997c into develop Nov 4, 2026
1 check passed
@ChristopherWirt

Copy link
Copy Markdown
Collaborator Author

think we're good - but we should clear out the bucket anyway of all the old files we've let everywhere

ChristopherWirt pushed a commit that referenced this pull request Nov 5, 2026
## Summary

Freezes `qc2026.11.0` in 0.91.0, the follow-up #85 left for when the
fork's first 0.91.0 build is cut. Until then the
edition is `immutable` but still a draft to `vortex-edition`, which
promises nothing about reading its files.

**Merge after upstream 0.91.0 is synced into develop**, so the first
build that records the freeze is a 0.91.0 one.

## Changes

- `min_library_version: Some("0.91.0")` on the list layout edition.
- `list_layout_edition_is_frozen` (was
`list_layout_edition_becomes_immutable`) also checks it isn't a draft.

## Testing

- `cargo test -p vortex-jni --lib`: 71 passed.

Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants