Repository navigation
Merge upstream develop into QC develop (2026-11-05) - #91
Merged
ChristopherWirt merged 42 commits intoNov 5, 2026
Merged
Conversation
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…a#10767) - Tracking Issue: vortex-data#10749 `DecimalArray::scalar_at` and the decimal `min_max` aggregate panicked when a stored value did not fit the precision of the dtype, because both built their results with `Scalar::decimal`, which calls `vortex_expect` on the precision check. Since `DecimalArray` construction doesn't check its values and file reads trust the stored precision, a file holding such a value could make a read panic. With this change both calls return an error instead. **Still open:** the panic is still reachable through cached stats. `min_max` returns a cached `Stat::Min`/`Stat::Max` before it reaches the kernel, and decimal stats decoded from a file are not checked against the precision, so an out-of-range stored stat still panics in `stat_scalar`. That needs a check at the read boundary and is tracked in vortex-data#10807. The other `Scalar::decimal` callers on compute paths (`sum`, `mean`, `cast`, `fill_null`, `between`, the DecimalByteParts kernels) are also left for follow-ups. --------- Signed-off-by: Vincent <1729065730@qq.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Remove `x8g.8xlarge` from the list of instance families that the dedicated benchmark runners can use. Signed-off-by: Robert Kruszewski <github@robertk.io>
## Summary - Tracking Issue: vortex-data#11114 `Date` accepts every `i32` day count and every `i64` millisecond count, but jiff only holds the years -9999 to 9999, so formatting a valid `Date` outside that range panicked. `Display` now prints the raw value and unit instead (e.g., `10000000days`), like the `Timestamp` fallback from vortex-data#10953. ## Changes Tests cover the first date after 9999-12-31, day and millisecond counts that a jiff `Span` can't hold, and an ordinary date whose output doesn't change. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…a#11183) ## Summary - Tracking Issue: vortex-data#11114 GeoArrow only allows nulls at the outer level of a geometry, but the validators of the list-based spatial types only checked the coordinate fields, so `LineString` also accepted `list({x=f64, y=f64}?)`. `LineString`, `MultiPoint`, `Polygon`, `MultiLineString`, and `MultiPolygon` now accept only the storage that their helper builds, at either top-level nullability. That leaves each type one storage form for each dimension, which the storage declarations (SD in vortex-data#11114) need. ## Changes Each validator compares the stored dtype with its storage helper, so the helpers stay the only definition of the form. `Point` and `Rect` already rejected nullable coordinate fields, and WKB has no nested level. Tests cover every nested level of each type, and reading a serialized `vortex.st.linestring` whose coordinate `Struct` is nullable. ## API Changes Building one of these five types with a nullable inner `List` or coordinate `Struct` (e.g., with `ExtDType::try_new`) now returns an error, and so does reading a dtype in that form. No Vortex writer produces that form, so only storage built by hand through the Rust API can hit it. Note that a file keeps its columns as fields of a struct whose field dtypes are parsed lazily, so reading such a column from a file currently panics instead of returning the error. A follow-up fixes that. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…x-data#10880) ## Summary Removes duplicate mutable-view slicing logic by making CudaViewMut::slice_mut delegate to into_slice_mut. The temporary copies only view metadata, not device data. The return lifetime keeps the original view mutably borrowed while the result lives. Bounds checks, panic messages, pointer arithmetic, and access tracking remain unchanged. ## Changes - refactor(cuda): slice mutable views through one implementation - docs(cuda): clarify mutable view borrow lifetime No performance impact is expected: this adds no allocation, CUDA calls, synchronization, or device-data copies. Optimized builds should eliminate the temporary view and inline the helper. No builds, tests, or benchmarks were run for this change. --------- Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
## Summary Corrects the documentation for release_file_mappings and vx_cuda_release_file_mappings. Keeping a file mapped does not pin its pages in memory: memory pressure can still reclaim them. On Linux, both posix_fadvise(POSIX_FADV_DONTNEED) and /proc/sys/vm/drop_caches skip pages mapped into a process. Release Vortex's cached mappings before using either for cold-cache measurements. Mappings still used by open readers remain mapped until those readers are released. ## Changes - Replace the claim that mapped pages cannot be evicted with the distinction between explicit cache eviction and memory-pressure reclaim. - Document both Linux eviction methods. - Include the memory-pressure caveat in both Rust and FFI documentation. - Regenerate the C header through cbindgen to match the FFI docs. Documentation only; no runtime behavior changes. No builds, tests, or benchmarks were run. --------- Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
…x-data#10881) ## Summary Reuses each test's existing host execution context when encoding inputs in the CUDA ALP and FOR tests, instead of creating a temporary context from the same session. This is test-only cleanup. Input values, assertions, and GPU execution contexts remain unchanged; production code is unaffected. ## Changes - Reuse the existing host context in five ALP tests. - Apply the same cleanup to the signed FOR and patched FOR tests. No builds, tests, or benchmarks were run for this change. --------- Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
…x-data#10884) ## Summary Adds CudaSlice::device_address for callers that only need an allocation's address, without ordering or recording a device access. Use it when building output-pointer metadata in FSST, OnPair, and Zstd, and for address checks in dynamic-dispatch tests. Actual GPU reads and writes remain tracked through launch arguments or device-pointer guards. ## Changes - Add and document the untracked device_address accessor. - Replace address-only device_ptr calls that immediately discarded their access guards. - Keep the tracked read in CudaDeviceBuffer::new: it orders the allocation stream after pending writes on other streams. Document why its guard can be dropped immediately. - Simplify the Zstd comments identifying where output writes are tracked. Each replaced lookup avoids two CUDA context checks, two access-state mutex acquisitions, and unnecessary read-tracking updates. These sites did not require cross-stream waits, so this removes bookkeeping rather than synchronization needed by the workload. No builds, tests, or benchmarks were run for this change. --------- Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
Signed-off-by: Frederic Branczyk <frederic.branczyk@dash0.com>
…ta#11005) Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…11090) With a session, the optimizer now works bottom up, and execution optimizes each node that it reaches. This prepares for lazy constructors in vortex-data#11091. ## Changes - `optimize_ctx` optimizes the children of a node that were never optimized before it tries the rules of the node. It does the same for the nodes that each rewrite builds. It does not visit marked subtrees, and it marks every array that it returns. Thus inverse rules, such as `slice(get_item)` and `get_item(slice)`, cannot loop. - One optimize call rewrites a shared node only once, for example the struct below every field of a projection. - `optimize_recursive` is now the same as `optimize_ctx`. - The session-less `optimize` still rewrites only the root and does not mark the result. Thus a later `optimize_ctx` still tries the session kernels. - `execute_until` optimizes the root before it starts, each child that execution focuses, the result of each `Done` step, and the array that it returns. Single-step execution optimizes with the session, instead of trying only the static reduce and parent-reduce rules. - `Chunked::reduce` does not reduce an empty variant array, because `Variant` has no empty canonical form. --------- Signed-off-by: Robert Kruszewski <github@robertk.io>
…ex-data#11154) Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…1200) Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
## Summary The count-based planner fills only 3 of the 5 fixed shard slots and keeps most packages together, although benchmark counts do not predict execution time. Balance recorded target runtimes across all 5 slots, splitting large packages at benchmark-target boundaries. The first CI run reduced the longest shard job from 10m03s to 5m07s. All 5 shards passed. Small packages stay together to limit repeated builds, and allocation-excluded targets remain separate. ## Changes - Record target durations from [develop run 41711616261](https://github.com/vortex-data/vortex/actions/runs/41711616261), commit `24cd0eac8f`, with a command to refresh the snapshot from simulation logs. Measure each complete binary execution, including startup and fixture construction, and use the median when several runs are supplied. - Allocate spare slots to the slower allocation-mode pool and assign the longest targets first. Split packages that exceed a shard's runtime budget or benchmark-count cap. - Preserve the 800-benchmark cap, allocation-mode separation, and complete benchmark binaries. A new target gets a five-second estimate and a warning until its runtime is recorded. - Add coverage for runtime balancing, package splitting, allocation-mode separation, count limits, deterministic output, and duration extraction. <details> <summary>Observed CI shard times</summary> | Shard | Reference develop run | This PR | | --- | --- | --- | | 1 | 10m03s | 4m44s | | 2 | 6m13s | 3m43s | | 3 | 2m43s | 4m42s | | 4 | 6s, unused | 5m07s | | 5 | 6s, unused | 2m27s | Compared [run 41711616261](https://github.com/vortex-data/vortex/actions/runs/41711616261) with [run 41716056593](https://github.com/vortex-data/vortex/actions/runs/41716056593) on `863c58f026`. These are elapsed CI job times including setup, builds, measurement, and uploads. Exact timings depend on cache and runner state. </details> CI passes the planner, all 5 simulation shards, Python formatting, linting, type checks, and all 58 script tests, including the 17 shard-planner tests. Other CI jobs are still running. Executed the planner locally to inspect the proposed assignment. Tests and linting were left to CI. Local hooks were bypassed under the requested CI-only workflow. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
- Spark columns read their values directly from the Arrow buffers: validity bits, fixed-width values, string offsets and string views. A decimal of 18 digits or fewer is returned as its unscaled long, as Spark's own vectorized readers do. - `ArrowProperties` turns off the bounds and null checks of the relocated Arrow on each value, as Spark does for its own Arrow, unless the user sets them. - Readers get the Hadoop configuration through a broadcast (`VortexIo.broadcast`), instead of with every task. - `IN` is pushed down as a native set instead of an `OR` of equalities. A null element stays in the set, so `NOT IN` keeps the SQL result. - The JNI writer takes ownership of the Arrow schemas it receives and releases them, so `writeBatch` no longer leaks the exported schema. The Java release added in vortex-data#11081 now runs only when native code fails before it takes the schema. - File statistics and zone maps skip variants nested in structs, lists and maps, which no builder can hold. Signed-off-by: Robert Kruszewski <github@robertk.io>
…ex-data#11182) ## Summary Since vortex-data#11173, sparse lists with a null or empty fill reuse the patch lists' elements in their output. When the patch lists are already zero-copy to a list, `MakeExact` only trims the ends, so null patch lists keep their elements. Those elements are exported and written. This PR uses `MakeExactWithEmptyNulls` on the null or empty fill path so that null patches are lists of length 0. ## Changes - The shared-elements path rebuilds the patch lists with `MakeExactWithEmptyNulls`. - If null lists hold nothing, the rebuild returns the array as is; otherwise the elements become a lazy filter over the ranges valid lists use. - `empty_fill_reuses_elements` test now uses a null patch with no elements. Signed-off-by: Matt Katz <mhkatz97@gmail.com>
…ortex-data#11192) Before: each benchmark job sent `POST /api/revalidate` and a warm pass of about 26 requests to bench.vortex.dev. During develop runs, this occurred about 15 to 60 times an hour. After: a reusable `refresh-bench-site.yml` workflow sends one revalidate and one warm pass. `develop-bench.yml` and `nightly-bench.yml` call it once per run, after all ingests, with `always()`. `commit-metadata.yml` calls it after its commit row. Thus a develop push gets two refreshes and a nightly run gets one. A failed revalidate is tried again three times. Ingestion stays in each job. The request shape agrees with `bench/CONTRACT.md`. Checks: `yamllint --strict`, ruff and pytest pass. A local test server showed the correct requests. `typos` was not installed. I read the workflow graph; no workflow run was started. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Will Manning <will@willmanning.io>
…a#11185) ## Summary - Tracking Issue: vortex-data#11114 A dictionary-encoded extension column imports today only because its importer declines the `Dictionary` type, after which `ArrowSession` retries with the value type. The next PR in this stack (SN2 in vortex-data#11114) makes the importers of known types fail on Arrow input outside their type instead of declining it, which would break these columns. So the session now peels the dictionary before any importer runs. The result is still a `Dict` over the imported values, so nothing is decompressed. ## Changes The existing dictionary test now also runs with an importer that fails on any dictionary. ## API Changes `ArrowImportVTable` implementations no longer receive `Dictionary` fields or arrays. They get the value field, with the dictionary field's metadata, and the values array. Note that a dictionary-encoded GeoArrow field with a CRS now keeps its CRS in the dtype from `ArrowSession::from_arrow_field` and `from_arrow_schema`, which matches what the array import already produced. Before, the name-only fallback of the spatial importers accepted the dictionary field itself and used default metadata. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…ask (vortex-data#11221) Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…uctor (vortex-data#9922) We fix cases where we were passing a vec that was immediately deallocated by passing an impl IntoIterator where possible For other cases we add a new constructor that leverages process_results in the caller and a size hint to avoid materialising list of items just to check the error case Signed-off-by: Robert Kruszewski <github@robertk.io>
…ortex-data#10147) FoR only compared for equality, so ordering comparisons decoded the array, which also bypassed the bit-packed child's own compare kernel. Encoding subtracts the minimum, so the encoded values are non-negative distances. For unsigned types the kernel compares them against the constant's distance directly; for signed types it does so when the child is bit-packed, which keeps the distances non-negative in the signed reading, and otherwise falls back to decoding. A distance that does not fit the type answers by which side of the reference the constant lies on. Signed-off-by: Robert Kruszewski <github@robertk.io>
Both read paths return `false` from `isSplitable`, so every Vortex file is one Spark input partition and a large file is read by one task. A dataset of a few large files can't use more tasks than it has files. - `isSplitable` is true for `.vortex` files on V1 and V2, so Spark plans byte splits as it does for Parquet, sized by `spark.sql.files.maxPartitionBytes`. - Each reader maps its byte range to a row range: an offset maps to the row at the same fraction of the file, rounded down to 8192 rows, the default writer's block. Neighbouring splits share an offset, so they share a row boundary and every row is read once whatever offsets Spark picks. The row count comes from the footer the reader opens anyway. Without an exact row count, the first split reads the whole file and the others read nothing. - A pushed `COUNT(*)` is not split, since it reads only the footer, and scan statistics read each split file's footer once. - On by default; `vortex.splitFiles=false` turns it off. A footer is read again on executors that haven't cached it, and a chunk that spans a cut is decoded by both splits next to it. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Mask tests test invariants of exact inputs and outputs. We do want to test object's observed properties of functions available in the public API. This is my attempt at replacing some of Mask's tests with ones based on Hegel. The new tests generate random inputs and check properties of operations on Masks still hold. This change also fixes operator== which worked incorrectly for empty Masks, returning false. --------- Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
vortex-data#11193) This PR draws the Explorer (vortex-web) with the Vortex design system and builds it with Bun. - The styles are copies of the system's CSS, plus `explorer.css` for the Explorer's own parts. Dtype colours come from the system. Encodings have no colour. - Blue marks only the selection. The header has links to the site, the docs, and bench. - Bytes use decimal units (1 KB = 1,000 bytes). The theme key is now `vx-theme`, and the old key moves over. - When a pane does not load, it says what did not load and what to do. The reason follows in a reference. - The PR removes parts that nothing used: LayoutSwimlane, AxisBar, SplitRegion, DtypeLegend, FileMap, and their styles. Not run: `bun run wasm`. Paired before and after screenshots will follow in a comment. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Will Manning <will@willmanning.io>
## Summary The snapshot used by vortex-data#11180 predates the benchmark cleanup merges, so it overestimates simulation counts and target runtimes. Refresh both snapshots from the five successful simulation shards in [develop run 41869576169](https://github.com/vortex-data/vortex/actions/runs/41869576169) at `141abdc590`. Note that these are just heuristics, so this will never affect correctness / running of benchmarks. ## Changes Recorded simulation cases fall from 1,869 to 1,408, and total target execution time falls from 711s to 400s. The refreshed plan estimates 83-84s for each regular shard and 65s for the allocation-excluded shard. These are target execution estimates, not full CI job times. Verified complete measurements for all 129 targets, exact target coverage, the 800-benchmark cap, and allocation-mode separation. The planner and `git diff --check` passed. No local tests or benchmarks were run. The pre-commit and pre-push hooks were bypassed as requested. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…0859) ## Summary - Tracking Issue: vortex-data#9971 `OperationsVTable::scalar_at` cannot retain preparation between row reads. The probe API introduced in vortex-data#9843 retains encoding state, validity, and child probes across repeated reads. This PR migrates 8 implementations: Bool, Constant, Decimal, Null, VarBinView, ByteBool, Sequence, PythonVTable. - These implementations use `ProbeState = ()`; generic probe validity is reused where needed. - `Constant` and `Null` scalars supply their own nullness - `Sequence` computes values directly - `PythonVTable` retains its unsupported scalar-access behavior. - Each migrated `scalar_at` remains a one-off probe compatibility shim. ### Verification - `cargo nextest run --cargo-profile ci --locked -p vortex-array -p vortex-bytebool -p vortex-sequence --all-features --no-fail-fast` — 4,832 passed, 1 skipped; the Python-binding Rust suite also passed (33 tests). - `uv run --locked --all-packages --reinstall-package vortex-data pytest --benchmark-disable vortex-python/test/test_pyarray.py vortex-python/test/test_scalar.py vortex-python/test/test_array.py vortex-python/test/test_encodings.py` — 86 passed, 1 skipped. - `cargo test --profile ci --locked -p vortex-array -p vortex-bytebool -p vortex-sequence -p vortex-python --all-features --doc --no-fail-fast` — 85 passed, 21 ignored. - `cargo +nightly-2026-09-10 fmt -p vortex-array -p vortex-bytebool -p vortex-sequence -p vortex-python --check` — passed. - `cargo clippy --profile ci --locked -p vortex-array -p vortex-bytebool -p vortex-sequence -p vortex-python --all-targets --all-features -- -D warnings` — passed. - Focused `cargo bench` runs — all 3 benchmark targets completed successfully. ## Comparison to baseline **Measured on a development machine (macOS with Apple Silicon); not representative of production performance** `cargo bench -p <crate> --bench <name> -- --min-time 0.1 --sample-count 15`. Values are Divan medians in microseconds per iteration, with one retained probe per iteration. | Array | Fixture | Reads | Baseline (µs) | Migrated (µs) | Change | | --- | --- | ---: | ---: | ---: | ---: | | Bool | nullable random | 256 | 4.957 | 4.624 | -6.7% | | Constant | fixed scalar, random | 256 | 2.666 | 2.103 | -21.1% | | Decimal | decimal128, random | 256 | 4.541 | 3.957 | -12.9% | | Null | all null, random | 256 | 1.958 | 1.260 | -35.6% | | VarBinView | nullable strings, random | 256 | 6.290 | 5.707 | -9.3% | | ByteBool | nullable random | 256 | 14.080 | 4.957 | -64.8% | | Sequence | i64 arithmetic, random | 256 | 8.582 | 2.333 | -72.8% | | PythonVTable | Scalar probing is unsupported by this vtable | — | N/A | N/A | N/A | ## AI Usage GPT-6-Sol implemented the changes with Codex. The human reviewed all the code and can explain every line of it. Benchmark results and the origin/develop comparisons are done by Codex. --------- Signed-off-by: Gunther Xing <jiengup@gmail.com>
Tracking Issue: vortex-data#10701 Top of the decimal `Widen` stack, on vortex-data#11005. No release gate. ## Summary A same-scale precision increase copied every value into a wider buffer even when the caller did not need the values yet. Returning a `Widen` defers that copy and lets subsequent kernels keep working on the narrow child. Basically, we might as well keep things in `Widen` form instead of casting immediately. ## Changes The child retains the source dtype and statistics because its values are unchanged. A cast of a decimal `Widen` passes to that child. Execution copies into the logical width directly, since yielding another widening cast would return a `Widen` again. The default-writer test compares round-trip values. Known cost: the retained benchmarks show slower filtering at precision 38 (see vortex-data#10946). <details> <summary>Historical local benchmark results</summary> These retained measurements predate this style pass and rebase. They do not measure the current head. `cargo bench -p vortex-array --bench widen_decimal`, Apple M4, medians. The numbers were measured at the top of this stack before the split, so they cover the whole stack, not this layer alone. The baseline is the canonical input, and the candidate is the same values as a `Widen` (child precision 5 under precision 10, 9 under 18, and 18 under 38). CodSpeed on this PR measures the layer itself. | bench | p10 canonical | p10 Widen | p18 canonical | p18 Widen | p38 canonical | p38 Widen | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | `cast_widen` | 12.08 µs | 6.958 µs | 12.33 µs | 6.958 µs | 16.54 µs | 14.58 µs | </details> ## API Changes A same-scale decimal cast returns a `Widen` when the target precision grows the canonical width. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…rtex-data#11233) ## Summary - Tracking Issue: vortex-data#11114 Follow-up to [the dictionary and run-end discussion in vortex-data#11185](vortex-data#11185 (comment)). ## Changes Explains why dictionary values need the outer field's extension metadata before importer dispatch, while run-end values have their own field and use normal nested-field imports. Documentation only. Tests, linting, and formatting were not run. Local commit and push hooks were skipped. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…tex-data#10575) ## Summary Speeds up the run-preserving path of the RunEnd filter kernel, most for short runs. The old loop counted the selected rows in each run with a separate `BitBuffer::count_range` call, and for short runs that call's setup cost more than the counting. ## Changes - **Rank lookup instead of range counts.** A filtered run end is the number of selected rows before the source run end. A small table of 64-bit mask words with running popcounts answers that with one load and one popcount per run. - **One pass.** The loop writes each new end at the current output position and moves forward only when the run has selected rows, so the ends come out compacted with no second pass. - **POPCNT through `fearless_simd`.** The x86-64 baseline has no `popcnt`, so the loop is dispatched with `dispatch!(Level::new(), …)`. Each level gets its own copy, which also lets `BitBuffer::collect_bool` inline. --------- Signed-off-by: Robert Kruszewski <github@robertk.io> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary `chunked_input_with_mixed_dimensions_returns_error` has failed on `develop` since vortex-data#10080 removed the "Mismatched types" text that it looks for. CI excludes `vortex-bench` from nextest, so CI never ran it. The conversion still rejects chunks with different vector dimensions at the `vortex_ensure_eq!` in `list_to_vector_ext`, so only the assertion was stale. ## Changes The test now checks that the error names both vector dtypes: dimension 2 from the second chunk against dimension 3 from the first. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…a#11238) ## Summary Every grouped `is_sorted` and `is_constant` call fails with "Return DType mismatch". Both `finalize` methods read the result field out of the nullable partial struct, which gives `Bool?` while the return dtype is a non-nullable `Bool`. vortex-data#11188 found the `is_sorted` case, and its grouped pushdown test covers only `min_max` because of it. ## Changes Both `finalize` methods now fill null partials. A null partial comes from an empty or null group and is the empty state, so each fill uses the verdict that `finalize_scalar` already gives that state: `true` for `is_sorted` (vacuously sorted) and `false` for `is_constant`. Grouped and scalar results now agree. Note that a null group gets the same verdict as an empty group, since the partial can't tell them apart and the return dtype is non-nullable. Each aggregate gets a grouped test that puts empty, all-null, and null groups next to ordinary ones. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
## Summary The DataFusion expression converter now pushes `array_sum` (`list_sum`) down to Vortex's `list_sum`. ## Changes - `array_sum(xs)` converts to `list_sum(xs)` with `include_nans`. DataFusion plans it as `array_sum(CAST(xs AS List<Float64>))`, and vortex-data#11161 pushes that cast down, so Vortex adds the same `Float64` elements in the same order. - Anything whose cast or sum Vortex cannot do (nested, fixed-size, decimal or string lists) stays in DataFusion. | Local A/B, pushdown vs DataFusion (time ratio) | geomean | |---|---| | 42 synthetic shapes (7 element types × list length 4/64/4096 × 0%/50% nulls) | 0.80 | --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>
## Summary This draft PR changes the look of the docs to the Vortex design system that vortex.dev, bench, and the Explorer use. It is a prototype for review. It does not change the content or the URLs of any page. ## Changes - `docs/_theme/vortex/system/` holds copies of the system's CSS (tokens, docs, and focus). `vortex_theme.py` stops the build if a copy has a hand edit. - `layout.html` writes the system's header, navigation, "On this page" list, and footer. The pages need no JavaScript, except for search. - Code blocks use the system's syntax colours, with a label for each language. - The Pygments accessibility styles, `accessible-pygments`, and the Breathe hook are removed. - At 390 px, no page is wider than the screen. Today the API page and the index are wider. The build passes with warnings as errors. After gzip, each page gets 0.9 to 1.9 KB larger, and the CSS goes from 2.7 KB to 5.4 KB. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Will Manning <will@willmanning.io>
## Summary Equality failures are easier to diagnose when errors include both values. Agent guidance should require `vortex_ensure_eq!` whenever possible so new equality checks include those diagnostics. ## Changes Added the rule to the Rust code style guidance in `AGENTS.md`. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…xplorer (vortex-data#11249) This change gives the docs and the Explorer the same chrome as vortex.dev and bench, and serves both from one Vercel project. - The docs and the Explorer show the shared header, footer, and theme switch. The theme is Dark by default, also when the OS uses Light. The vx-theme cookie keeps the choice on all vortex.dev subdomains. - Both show one 404, with a 404 status. The docs build it, and it is the same as the 404 on vortex.dev. The Explorer has no 404 of its own now. The Explorer's start page shows the dot field. - The docs' Vercel project deploys the Explorer. A preview serves it at `/explorer/`. explore.vortex.dev serves it at the root, and docs.vortex.dev/explorer redirects there. After merge, move the explore.vortex.dev domain to the docs' Vercel project. Until then, it serves its last deploy. Not in this change: one favicon set, one share image, and the GitHub star count. ## Screenshots Before is production. After is a local build. <details><summary>Docs</summary> Dark, 1440px  Dark, 390px  Light, 1440px  Light, 390px  </details> <details><summary>Docs footer</summary> Dark, 1440px  Light, 1440px  </details> <details><summary>Docs 404</summary> Dark, 1440px  Dark, 390px  Light, 1440px  Light, 390px  </details> <details><summary>Explorer start page</summary> Dark, 1440px  Dark, 390px  Light, 1440px  Light, 390px  </details> <details><summary>Explorer file view</summary> Dark, 1440px  Dark, 390px  Light, 1440px  Light, 390px  </details> <details><summary>Explorer 404</summary> Dark, 1440px  Dark, 390px  Light, 1440px  Light, 390px  </details> 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Will Manning <will@willmanning.io>
… cached data generation (vortex-data#11244) ## Summary The local random-access benchmark job took ~23 min: 18.3 min of timed measurements and 4.1 min of data generation, ~80% of which was writing LitData. This PR cuts the job to roughly 12 min without changing what is measured. ## Changes - **5 s time limit per measurement** (was the 10 s default). `scripts/random-access-split.py` passes `--time-limit 5`. Each number is a median per-lookup latency, so the time limit only sets the sample count. Even at 5 s the fast formats take thousands of samples. - **Parallel data generation.** `--prepare-data` runs one process per dataset concurrently. Each dataset's output is printed in its own log group. - **Cached Parquet and LitData data.** `develop-bench.yml` and `pr-bench-runner.yml` restore Parquet and LitData with `actions/cache`. Vortex, Arrow IPC and Lance are written every run, so the formats under test always come from the current code and lockfile. The key covers the dataset generators (`vortex-bench/src/datasets/**`, `vortex-bench/src/random_access/**`), the `parquet`/`arrow-*`/`rand*` versions in `Cargo.lock`, the LitData script, `litdata.rs` and the pinned litdata version. Local and S3 jobs use separate keys. --------- Signed-off-by: Claude <noreply@anthropic.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
) Tracking Issue: vortex-data#9128 ## Summary Combines planning and execution for eligible RowFn batches in one type dispatch. This is the shared framework layer for vortex-data#10676, which contains the primitive comparison migration. It builds on vortex-data#10899, which separates batch inspection from planning and adds the regression cases this PR must keep passing. ## Changes - Decodes a direct Boolean batch straight from the borrowed inputs, after `BatchProbe` has classified all of them, so a later null literal still skips an earlier fallible expression or device transfer. Fallible decoders keep valid-row execution. - Finishes all-null, all-constant, null-free, and dense batches in the planning visit. Every other batch comes back as a `PartiallyValidBatch`, the only input that dense retry and valid-row execution accept, so every caller runs the fast paths first. - Constructs Boolean output with its validity in the shared executor. Dense execution masks any output that does not carry that validity, and a validity of the wrong length returns an error. Primitive-specific CPU and lane-width packing policies stay out of this layer. - In builds that enable AVX-512F and AVX-512BW at compile time, `collect_bool_words_multiversioned` uses the inline packing loop instead of the `#[target_feature]` copy, so its inlining no longer depends on codegen-unit placement. ## Performance This PR lowers the fixed cost of each RowFn call, so CodSpeed shows no gain for it alone. The gain shows up in vortex-data#10676, which moves primitive comparison onto the RowFn executor. To isolate this PR, vortex-data#11220 applied the two vortex-data#10676 commits directly on `develop`. Without this PR, comparisons on small arrays are 10% to 16% slower than `develop` in CPU simulation, including the zone map pruning benchmarks, `i64_lt`, and `timestamp_gte`. The added time barely changes between 16 and 1,024 zones (18.0 µs and 17.3 µs for `int_gt`), so it is paid per call, not per row. On top of this PR, vortex-data#10676 shows none of these regressions, and its walltime comparison gains are larger: `compare_u64_avx2` improves by 31% instead of 16%, and seven more comparison benchmarks improve by 12% to 18%. In the AVX-512 build, the RowFn changes moved the deferred Boolean kernel's AVX-512 packing loop out of its caller's codegen unit, so LLVM could no longer inline it. CodSpeed reported the two partially valid `deferred_bool_avx512` cases as 42% and 16% slower. With the last commit, CodSpeed no longer flags either case. Builds that detect AVX-512 at runtime compile the same code as before. vortex-data#11220 also shows `compare_u8_avx512` and `compare_f32_avx512` as 14% and 10% slower without this PR, which matches the same packing problem, but that build's assembly wasn't inspected. CodSpeed on the current head flags three varbin comparison cases and `random_i8[0.01]` as 12% to 18% slower, and one `take_filter` case as 13% faster. None of them runs a RowFn. vortex-data#10676 reverses the varbin and `take_filter` changes without touching their code, so they follow code placement. `random_i8[0.01]` isn't a stable signal either: its `develop` baseline moved from 99.6 µs to 119.3 µs between `bc58587` and `c9a3fbe`, and vortex-data#11220 moved it 13% the other way. <details> <summary>CodSpeed results for vortex-data#10676 with and without this PR</summary> The first column is vortex-data#11220, which compares the vortex-data#10676 commits on `develop` with `develop` (`c9a3fbe`). The second column is the vortex-data#10676 report, which compares the same commits on this PR with this PR (`b0a64dd`). Against `develop` (`bc58587`), this PR flags none of these benchmarks. CodSpeed marks the walltime runners as unreliable, and the two columns come from different runs. | Benchmark | Mode | Without this PR | With this PR | | --- | --- | ---: | ---: | | `int_gt[16]` | CPU simulation | 15.6% slower | not flagged | | `int_gt[1024]` | CPU simulation | 14.5% slower | not flagged | | `float_gt[16]` | CPU simulation | 12.2% slower | not flagged | | `float_gt[1024]` | CPU simulation | 12.1% slower | not flagged | | `or_chain[16]` | CPU simulation | 12.3% slower | not flagged | | `or_chain[1024]` | CPU simulation | 11.9% slower | not flagged | | `is_not_null_pred[16]` | CPU simulation | 11.6% slower | not flagged | | `is_not_null_pred[1024]` | CPU simulation | 10.5% slower | not flagged | | `i64_lt` | CPU simulation | 11.7% slower | not flagged | | `timestamp_gte` | CPU simulation | 10.3% slower | not flagged | | `bench_compare_sliced_dict_primitive[(1000, 10000)]` | CPU simulation | 11.5% slower | not flagged | | `encode_primitives[f32, (2000, 2)]` | CPU simulation | 12.8% slower | not flagged | | `compare_u8_avx512` | Walltime | 14.4% slower | not flagged | | `compare_u8_neon` | Walltime | 13.3% slower | not flagged | | `compare_f32_avx512` | Walltime | 10.4% slower | not flagged | | `compare_float_avx2` | Walltime | 18.4% faster | 28.2% faster | | `compare_u64_avx2` | Walltime | 15.7% faster | 30.8% faster | | `lt_i64_nullable_avx2` | Walltime | not flagged | 18.4% faster | | `compare_int_constant_neon` | Walltime | not flagged | 16.2% faster | | `compare_int_eq_avx2` | Walltime | not flagged | 15.4% faster | | `compare_f32_avx2` | Walltime | not flagged | 15.4% faster | | `compare_int_constant_avx2` | Walltime | not flagged | 15.3% faster | | `compare_int_avx2` | Walltime | not flagged | 13.5% faster | | `compare_int_nullable_avx2` | Walltime | not flagged | 12.5% faster | </details> <details> <summary>AVX-512 investigation on EC2</summary> C7i metal (Xeon Platinum 8488C) with the CodSpeed AVX-512 build flags, the bench profile, one isolated CPU, and 21 paired blocks of 1,000 samples. Ratios are the median paired ratio to `develop` (`8c87c4c457`). Below 1 is faster. An A/A comparison of two identical binaries flags none of the 100 cases. | `deferred_bool_avx512` case | `develop` | Before fix | With fix | | --- | ---: | ---: | ---: | | `ConstantLhs, PartialAccepted` | 32.1 µs | 1.354 | 1.025 | | `Columns, PartialAccepted` | 37.7 µs | 1.159 | 1.018 | Before the fix, the kernel executed 15% more instructions per row. LLVM inline remarks show `collect_bool_words_avx512` inlined on `develop` and "definition is unavailable" on this branch. Across all 100 `#[cpu_features]` cases, the fix also returns `compare_string_lt` to the `develop` speed. It makes two `intersect_by_rank` cases and one `starts_with` case 3.5% to 5.6% slower, but those execute the same instructions as before, so the difference is code placement. The AVX2 build is unchanged. </details> <details> <summary>Historical EC2 framework measurements</summary> These numbers belong to an older framework replay and do not establish the performance of this revision. Ratios are the median of 21 paired candidate/baseline elapsed-time ratios. Below 1 is faster. This compares the framework alone, F `94be28140d5bd7e6b654af5e462253add2e49044`, with D `0291778db36809f77ce0219bf9850506d3cba3b6`. | Workload | C7g NEON | C6a AVX2 | C7i AVX2 build | C7i AVX-512 build | | --- | ---: | ---: | ---: | ---: | | `u64` multiply, 8,192 rows | 1.732 | 0.979 | 0.992 | 0.998 | | Nullable `i64` add, 32,768 rows | 1.006 | 1.004 | 1.154 | 1.005 | The actual framework pair order was imbalanced: D ran before F in 16 blocks and after it in five. Treat these figures as diagnostic observations, not reliable causal effect sizes. EC2 C7g (Neoverse V1), C6a (EPYC 7R13), and C7i (Xeon Platinum 8488C). Rust 1.99.1 / LLVM 23.1.1, bench profile with 16 CGUs and no LTO, MiMalloc, CPU 2 pinned. The same 455-case public-path microbenchmark suite ran serially across hosts, with 21 paired blocks, 10 ms per case, A/A calibration, and a separate control executable. These are microbenchmarks, not query results. C7i can dispatch to AVX-512 at runtime even in its AVX2 build. </details> The separately confirmed NEON comparison gains were published in vortex-data#10843, which has merged. They do not measure this framework. CI passed on `b0a64dd`, including all-target, all-feature Clippy with warnings denied and the Linux, Windows, sanitizer, and Miri test jobs. The CodSpeed check fails on the flags described above. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
gitRasheed
approved these changes
Nov 5, 2026
This was referenced Nov 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge upstream
vortex-data/vortex:developatd8a4952130ea6f5871f11e2da5302725c3cfce5finto QCdevelopat2eff49f093b3a99a7aba68c466ac1281b11825cc, bringing in 41 upstream commits.The conflict resolutions preserve QC's split-concurrency controls, session defaults, schema-evolution handling, and broader file-splitting tests. They incorporate upstream's Hadoop configuration broadcasts, Arrow validity and string-view optimizations, and null-aware
INsemantics. Non-nullINlists retain QC's zone-pruning bounds; lists containing null use the upstream native set without bounds so negation preserves SQL null semantics.Reviewed the conflict resolutions and confirmed the merge has no unresolved paths. Builds, tests, formatting, and linting were not run locally, per repository guidance; CI will run on this PR.
The merge commit is signed off as Christopher Wirt. Upstream history is preserved; upstream commit
c15a41d13a72ae28683d87406f2d50a68de36fa7(pin Zig to 0.17.0) has no sign-off trailer.