Repository navigation
Conversation
…mat#8954) ## Summary - correct `Dict[str, any]` to `Dict[str, Any]` - parameterize public dictionaries and `AbstractContextManager` - use the directly imported Arrow `Expression` type instead of an unresolved module attribute - annotate variadic keyword arguments and the `None` return from `create_scalar_index` Co-authored-by: Xuanwo <github@xuanwo.io>
…ance-format#7465) ## Summary The structural decoder's `StructuralFieldScheduler::initialize` eagerly initializes **every** page in a column before scheduling, even when the read only touches one page. For a column with thousands of large, non-coalescing pages (e.g. a BTree leaf scan that reads one leaf), this means paying metadata IO proportional to the page count regardless of how few rows are actually requested. This PR makes page-metadata initialization **range-scoped and per-page**: - `DecodeBatchScheduler::try_new` and `StructuralFieldScheduler::initialize` now take an optional `requested_ranges: Option<Arc<[Range<u64>]>>` — the top-level row ranges that will later be scheduled. The structural path initializes only the pages those ranges overlap (`pages_overlapping_ranges`). `None` preserves the prior eager behavior. - The page-scheduler `initialize` is split into `init_ranges()` (declare the byte ranges the page needs) and `init_from_buffers(buffers, io)` (finish initialization from those bytes). The field scheduler concatenates the misses' ranges into a **single** `submit_request` so adjacent ranges coalesce into shared GETs instead of one request per page. - Page metadata is cached per page via `PageDataCacheKey { column_index, page_index, view_tag }` (replacing the per-column `FieldDataCacheKey`). `view_tag` is retained so a column decoded under two shapes (blob descriptor `Struct<pos,size>` vs raw `LargeBinary`) can't collide on cached state. - Nested schedulers forward ranges consistently with their existing `schedule_ranges`: lists/maps/structs pass the top-level ranges through unchanged; fixed-size-list scales by `dimension`. The requested ranges are always a **superset** of what `schedule_ranges`/`schedule_take` later touch, so any page a read schedules was initialized here. Net effect: a cold point/range read's metadata IO is invariant to the column's total page count. ## Test plan - [x] `cargo test -p lance-encoding --lib` (393 passed) — the round-trip encoding harness now exercises **both** the eager path (full scan, `None`) and the lazy path (range reads and scattered takes pass their `requested_ranges`). - [x] New unit test `test_initialize_coalesces_missed_page_metadata`: an N-page cache miss issues exactly **one** `submit_request` carrying every page's metadata range (not one request per page). - [x] Updated FullZip page-scheduler tests to the `init_ranges` / `init_from_buffers` split. - [x] `cargo test -p lance-file --lib` (76 passed). - [x] `cargo test -p lance --lib dataset::blob::tests::test_blob_cache_key_distinguishes_views` — confirms the per-page key still distinguishes blob decoder views. - [x] `cargo fmt --all` and `cargo clippy -p lance-encoding -p lance-file --tests --benches -- -D warnings` clean. Made with [Cursor](https://cursor.com) --------- Co-authored-by: Cursor <cursoragent@cursor.com>
Both parents have landed — lance-format#9160 squash-merged into `main`, carrying lance-format#9167 with it — so this is no longer stacked. Rebased onto `main`; the diff is now this PR's own commit alone. ## The gap `plan_search` took one column and bound the whole query to it (`FtsQuery::with_column` rebinds *every* leaf, including a multi-match's per-column ones). So a cross-column multi-match was rejected: the memtable holds one inverted index per column and there was nothing to route it to. ## Decompose, don't thread A top-level multi-match splits into one per-column match, each planned by the existing single-column path, and the results are unioned. That was the design call worth making. Threading a column dimension through `plan_search` / `build_source_plan` / granularity resolution would have touched every per-source behavior — prefilter, the cross-generation block-list, the transient-index fallback from lance-format#9167 — and risked each of them diverging between the one-column and many-column cases. Decomposing reuses all of it verbatim. The cost is one pass over the sources per column. None of the base-table cross-column machinery was reusable here, for the record: `cross_column_compound_search` (2,147 lines) operates on `Vec<Arc<InvertedIndex>>` — on-disk indexes — and the memtable has `FtsMemIndex` with a separate evaluator. ## Collapsing the duplicates A row matching in two columns is scored twice and arrives twice. The base-table path scores each field independently and takes the **maximum** per row (`DisjunctionScore::Max`), and over an input already sorted by `_score` descending, keeping the first occurrence per primary key *is* that maximum. `FirstByPkExec` does that — a filter, not a grouped aggregate, so the streaming top-k shape the FTS planner is built around survives. Its input ordering is load-bearing and the module doc says so. ## What stays refused Only multi-match decomposes. A boolean or boost spanning columns is one predicate over several fields — `must: [a in title, b in body]` is a conjunction, not a union of per-column results — so unioning per-column plans would answer a different query. Refused, with the reason in the message. ## Tests - `multi_match_searches_every_column_and_collapses_duplicates` — four memtable rows: title-only, body-only, both, neither. Asserts all three matching rows come back **and** that the row matching in both appears exactly once. - `cross_column_boolean_is_refused` — the non-decomposable shape. - Supersedes `multi_match_query_is_refused_by_the_active_memtable` from lance-format#9167; that restriction is what this lifts. 714 `mem_wal` tests pass on the rebased tree. (The count grew from 703 because `main` picked up `mem_wal` tests from other merged work — this PR adds two.) ## Note for the sophon side The max is taken over per-source **local** BM25 scores, so a cross-column query on a WAL table can rank differently from a non-WAL table for two reasons rather than one. That widens the existing local-scoring trade rather than introducing a new class of divergence — but it is worth knowing when the sophon multi-match arm lands. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01QBv2WLSq5oy6kR2gVrfFDh --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ance-format#9106) ## What this adds This PR defines the common read contract for one fragment-rewrite mapping: ```rust async fn remap_row_id(&self, row_id: u64) -> Result<Option<u64>>; async fn remap_row_ids( &self, row_ids: &[u64], ) -> Result<Vec<Option<u64>>>; ``` A row ID is a physical row address containing a fragment ID and row offset. A mapped row ID identifies the rewritten destination row; `None` means the source row was deleted. `remap_row_id` is the required primitive. The default batch implementation calls it for each input in order. Mapping implementations can override the batch method to coalesce IO or translate more efficiently. The contract requires batch results to preserve input order and duplicates and return one result per input. Addresses outside the mapping's source fragments return `InvalidInput`; corrupt persisted mappings return `CorruptFile`; storage errors propagate unchanged. ## Ordered compaction `OrderedCompactionMapping` adapts the existing `RowAddrRemap` bitmap/rank implementation to this contract: ```text source physical row ID │ ▼ existing RowAddrRemap │ ├── destination physical row ID └── None when deleted ``` It reuses the existing compaction mapping and validated fragment layout without duplicating source or destination metadata. The legacy FRI reader and writer remain unchanged. This interface represents one mapping only. FRI lineage traversal, coverage derivation, and planner integration remain responsibilities of the common FRI reader. Stable-partition mappings implement the same interface in lance-format#9064. ## Validation Tests cover default batch delegation and error propagation, deleted rows, duplicate and out-of-order inputs, empty batches, single/batch equivalence, invalid source fragments, and out-of-range source offsets. Validated with workspace Clippy and `cargo fmt --all`.
…format#9268) Writing a nullable struct inside a list can lose the repetition/definition slots reserved for null and empty lists. Recording the struct validity resets `current_len` to the child array length, so recording its nullable child next violates the serializer length invariant. Keep the special slots in `current_len` when recording validity. The regression round-trips a null list, a populated list with null struct and child values, and an empty list through both validity layers and the list offsets. The required `cargo clippy --all --tests --benches -- -D warnings` is blocked by existing `clippy::single_range_in_vec_init` diagnostics in `encodings/logical/primitive/blob.rs`, `array_encoding/physical/bitpack.rs`, and `encodings/logical/primitive/sparse.rs` on Rust 1.97.0.
…e-format#8927) close lance-format#8857 Apply the scanner's `batch_size_bytes` budget a second time after blob v2 payloads are materialized, splitting the merged batch by bytes so blob reads are capped at 2x oversubscription relative to the inline/descriptor budget. --------- Co-authored-by: Xuanwo <github@xuanwo.io>
…es (lance-format#8985) ## What this changes `.cargo/config.toml` and `clippy.toml` join `python.yml`'s `pull_request` `paths:` filter. Two lines, no job changes. Closes lance-format#8906. ## Why the root `.cargo/config.toml` reaches the Python builds Cargo merges configuration up the directory hierarchy, so a build rooted in `python/` reads both `python/.cargo/config.toml` and the repository-root file. Measured, because the interesting part is what happens to a key both files set: with `[target.aarch64-apple-darwin] rustflags = ["--cfg", "outer_flag"]` in a parent `.cargo/config.toml` and `["--cfg", "inner_flag"]` in the child's, `cargo build -v` in the child passes both to rustc. Arrays are joined rather than shadowed, so a root-only edit to either `rustflags` line changes what the Python extension compiles with even though the sibling sets the same key. On top of that, `[profile.release-no-lto]` and `[profile.bench]` are defined only in the root file, so the Python builds take them from there outright. ## Why `clippy.toml` as well `python.yml`'s `lint` job runs `cargo clippy` with `working-directory: python`, and clippy reads an ancestor `clippy.toml`. Measured the same way: a crate in a subdirectory of a directory holding a `clippy.toml` with an unknown field fails with `error reading Clippy's configuration file: unknown field`. The repository has exactly one `clippy.toml`, at the root, and no `python/clippy.toml`, so that file is what this job's clippy run uses. Its `disallowed-macros` entries gate the Rust code behind the Python extension, and editing them currently triggers no Python job. ## Test plan `yaml.safe_load` parses the file and returns the filter with the two new entries in place. The trigger itself is not exercisable from inside this pull request: proving the new entries match would take a pull request whose only change is one of those two files. What this one does show is the existing behaviour, since it edits `.github/workflows/python.yml`, which the filter already lists, so the Python jobs run here. `rust.yml` got the same two entries in lance-format#8867. Co-authored-by: Xuanwo <github@xuanwo.io>
…ainers (lance-format#8996) ## Problem The three streaming IVF normalization sites called `convert_to_floating_point`, which returns f16 and f64 unchanged and only widens integer types. Those columns reached the trainers' unconditional Float32 downcast and panicked the build. Fixes lance-format#8995. ## What this changes All three sites now go through one helper that casts the chunk to Float32. Two details worth calling out. An f64 above `f32::MAX` saturates to an infinity rather than failing the cast. The samplers run their finite filter before the cast (ivf.rs:3193 and :3518), so without something else the trainers could receive infinities, and `lance-index/src/vector/kmeans.rs:840` has no finite guard: the centroids would come out inf or NaN with no error. `drop_rows_that_saturated` removes exactly the rows the narrowing broke and logs how many, so a null row still passes through the way it does for every other value type. The helper takes the array apart with `into_parts` rather than rebuilding the field by hand, which keeps the inner field's metadata and drops an unreachable branch. Its accept list is f16/f32/f64/i8, matching what a vector column can actually hold (rust/lance/src/index/vector/utils.rs:244). The `num_partitions <= 256` streaming path dispatches on value type already and is untouched. Note for whoever reviews lance-format#8610: that PR moves this same `convert_to_floating_point` call into a new helper without changing it, so it carries the panic forward. Landing this first, or folding it in, avoids that. ## Test plan Five unit tests on the helper: the dtype matrix compared value by value, a sliced chunk mapping to the rows it points at, nulls staying on their row, an f64 that overflows f32 being dropped while a null row beside it survives, and an unsupported type returning `InvalidInput` with the type named. Removing the saturation filter makes the overflow test report `[[1.0, 2.0], [inf, 4.0], [5.0, 6.0]]`. Two end-to-end cases train a 257 partition model with streaming enabled, one f16 and one f64. They take different routes on purpose: a non-nullable column gets the fixed-range sampler, a nullable one the resampling path, and each has its own normalization site, so the two cases cover two of the three sites rather than differing only in dtype. Restoring the old normalization at the coreset site makes both panic in arrow's cast. - `cargo test -p lance --lib` 3387 passed, 3 ignored - `cargo test -p lance --lib index::vector::ivf::tests::` 51 passed - `cargo clippy --all --tests --benches -- -D warnings` clean - `cargo fmt --all --check` clean --------- Co-authored-by: Xuanwo <github@xuanwo.io>
…umn (lance-format#8999) ## Problem The PQ transform derives the sub-vector width from the column dimension, so a caller-supplied codebook of the wrong size is not a local error. An oversized codebook has its sub-vector boundaries read at the wrong offsets and builds a searchable but wrong index without complaining; an undersized one slices out of bounds and panics. Fixes lance-format#8998. ## What this changes `validate_supplied_codebook` in lance-index rejects a codebook whose length is not `num_centroids * dimension`, and reports the size as not representable instead of wrapping for `num_bits` of 0 or something absurd. Three sites consume a supplied codebook and all three now call it: the v3 quantizer build, the legacy IVF_PQ writer (`build_pq_model_in_fragments`), and the distributed global PQ (`make_global_pq`, where a codebook is mandatory). Putting the check at a language boundary instead would leave the Rust and Java callers unguarded, since `PQBuildParams::with_codebook` is public Rust API. ## Test plan `test_validate_supplied_codebook` covers the rule directly: the correct size for 8 and 4 bits, oversized, undersized, a dimension that does not divide by `num_sub_vectors`, `num_bits = 0`, and `num_bits = 1024`. `test_create_ivf_pq_rejects_mismatched_codebook` drives a real `create_index` for both index file versions, which reach the codebook through different writers. Negative controls: with the core check removed the v3 case succeeds and writes `index.idx` (586 bytes) plus `auxiliary.idx` (71803 bytes); with the legacy writer's check removed the legacy case succeeds and writes `index.idx` (77908 bytes). Undersized panics at pq.rs:92 with `range end index 6144 out of range for slice of length 4096`. - `cargo test -p lance --lib` 3382 passed, 3 ignored - `cargo test -p lance-index --lib vector::pq::tests::` 5 passed - `cargo clippy --all --tests --benches -- -D warnings` clean - `cargo fmt --all --check` clean ## Not in this change The IVF centroid width is unvalidated too, but a mismatch there already errors rather than building a bad index: `IVF centroids length mismatch: 2 != 64` (rust/lance/src/index/vector/ivf.rs:1696). That message compares a row count against a value count, so it is worth fixing, separately. --------- Co-authored-by: Xuanwo <github@xuanwo.io>
…t#9276) `open_readers_impl` opened a fragment's data files one after another. Each open is at least one object-store round trip for the file's tail metadata, so a fragment that has accumulated several data files (one per `add_columns` / merge) paid the round trips back to back before its first byte of data could be requested. Open the files with `try_join_all`, which keeps the readers in data-file order; the rest of the function is unchanged. Measured with a chrome trace of a take over 4 fragments x 4 data files (a large stable-row-id table on S3, ~25 ms per GET at the time): the 16 tail reads used to arrive in four waves at +0, +25, +49 and +75 ms and the first data read started at +164 ms; now all 16 start within 1 ms and the first data read starts at +49 ms. End-to-end take/scan wall time is within network noise (+-150 ms) on this table because fragment-level parallelism already overlaps most of the latency; the gain is per-fragment open latency, which is what bounds time to first batch.
…at#9277) `len` counted the holes with `holes.iter().count()`, walking every hole through a `Box<dyn Iterator>` with a virtual call per element, although `EncodedU64Array::len` is O(1). `RowIdIndex::new` calls `len` for every segment (lookup build, `start_offset`, `merge_rows`), so on a table with 3,488 `RangeWithHoles` segments holding ~9 million holes this showed up as ~5% of the index build in `perf`. Benchmark (a large stable-row-id table, 20,760 fragments, `RowIdIndex` build, cold / second build, `--profile bench`): glibc malloc default: 745 / 353 ms -> 665 / 300 ms glibc.malloc.hugetlb=1: 314 / 291 ms -> 265 / 234 ms
…-format#9278) ## Problem Every `read_stream`/`read_range` on a structural (2.1+) file rebuilt a boxed `StructuralPageScheduler` for **every page of every projected column** in `StructuralPrimitiveFieldScheduler::try_new`, cloning buffer tables and instantiating decompressors. lance-format#7465 made page *initialization* range-scoped, but construction stayed O(pages). For a column with hundreds of thousands of pages this is a large fixed cost per read, which is exactly the pattern IVF prewarm follows (one read per partition window; 4,069+ windows over a 620,881-page `auxiliary.idx` in the September 10B RQ benchmark) and that nprobes=1 queries pay per query. The captured CPU profile was dominated by malloc/free of `PageInfoAndScheduler`/`MiniBlockScheduler`. Linear: [OSS-2276](https://linear.app/lancedb/issue/OSS-2276) ## Change - `try_new` now stores only the per-page row boundaries (`page_row_ends`, one `u64` per page) plus a `PageSchedulerSource` (column info, decompression strategy, target field). - `initialize(Some(ranges))` builds schedulers only for the pages those ranges touch, merged in page order; `initialize(None)` keeps the eager behaviour and builds every page. - Range-to-page mapping works from `page_row_ends`, so it no longer needs built schedulers. - Scheduling a page that was not covered by the initialized ranges returns `Error::Internal` with a descriptive message instead of panicking on an uninitialized page. - The lazy scheduler must own its column info and decompression strategy, so the new constructor is `try_new_lazy(Arc<ColumnInfo>, Arc<dyn DecompressionStrategy>, ..)`. The existing `try_new(&ColumnInfo, &dyn DecompressionStrategy, ..)` keeps its signature and eager behaviour and is marked `#[deprecated]`. ## Performance Matched local micro-benchmark against `main` (3a12096): a v2.1 file with 5 `UInt8` columns, 131,072 pages per column (655,360 pages total), read from local disk, reading successive disjoint row windows with `read_stream`, 200 reads per pass, two repetitions per binary. Both binaries built from the same harness with `cargo build --release`; numbers below are the second repetition's warm pass (page metadata cached), lower is better. Machine: Apple Silicon Mac, single process. | Scenario / metric | Baseline | This PR | Benefit | | --- | ---: | ---: | ---: | | 64K-row read (256 pages/column touched), mean latency | 83.0 ms | 19.5 ms | 4.3x speedup | | 64K-row read, p99 latency | 106.5 ms | 22.5 ms | 4.7x speedup | | 4K-row read (16 pages/column touched), mean latency | 68.0 ms | 7.8 ms | 8.7x speedup | | 4K-row read, p99 latency | 79.2 ms | 11.2 ms | 7.1x speedup | Cold-pass numbers (first touch of each window's page metadata) show the same ratios. The baseline's ~68 ms floor is the whole-file scheduler construction; this PR removes it. ### End-to-end: prewarming one 1B IVF_RQ segment Same index as the Linear issue (RQ1 1-bit, 244,140 partitions, 1B rows, `auxiliary.idx` 113.8 GB), copied to local EBS on an r8i.8xlarge (32 vCPU, 247 GiB RAM), `LANCE_CPU_THREADS=30`, 192 GiB index cache, both wheels built from source with `--profile release`. Cold = page cache dropped before the run. Lower is better for time, higher for fill rate. | Scenario / metric | Baseline (`main` 3a12096) | This PR | Benefit | | --- | ---: | ---: | ---: | | Cold prewarm of 118.0 GB, 64 MiB windows, wall time | > 3,600 s (timed out at 104.4 GB, 88%) | 153.8 s | > 23x speedup | | Cold prewarm, average cache fill rate | 29 MB/s (decaying from 77 to 15 MB/s) | 767 MB/s | 26x | | Cold prewarm, process CPU | 1 core at 100% for the whole hour | 285 CPU-s over 154 s | n/a | | Warm page cache prewarm, 64 MiB windows | not run (baseline too slow) | 61.1 s (1,933 MB/s, 0 physical reads) | n/a | | Warm page cache prewarm, 1 GiB windows (issue config) | not run | 60.9 s | n/a | | 64 nprobes=1 queries without prewarm (partition loads on demand) | 19.0 s | 6.9 s | 2.8x speedup | Peak RSS after prewarm was 121 GB on this PR (cache 118.0 GB). The 64 nprobes=1, k=10 query results (row ids and float32 distances) hash identically on `main` and this PR, both with and without prewarm. Raw summaries and 5-second progress logs: `s3://mmlb-us-east-1/rq-benchmark/results/20260916-oss-2276-prewarm-ab/`. **Not yet verified:** QPS under load on the 10B index. The remaining ~7.7 ms fixed cost per read at this page count is two O(pages) walks over immutable `ColumnInfo` (the `page_row_ends` prefix sum here and `normalized_column_num_rows` in lance-file's read-length validation); caching those per file is a follow-up. ## Validation - New test `initialize_builds_only_requested_page_schedulers`: construction builds nothing; range-scoped initialize builds exactly the touched pages in order; scheduling an uncovered page errors instead of panicking; widening the request adds only new pages; `None` builds all and drops the source. - Existing lazy-initialization tests adapted to the new constructor. - `cargo test -p lance-encoding` (1826 passed), `cargo test -p lance-file` (182 passed), `cargo test -p lance --lib ivf` (277 passed), `cargo check --workspace --tests --benches`, `cargo fmt --all -- --check`, `cargo clippy --all --tests --benches -- -D warnings`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…-format#9001) ## Problem The numpy `ivf_centroids` branch compared the centroid count against `num_partitions` in the same condition as the shape check. `num_partitions` defaults to None and is documented as deprecated in favor of `target_partition_size`, so a valid 2D array supplied without it failed, reported as "Ivf centroids must be 2D array" against a 2D shape. Fixes lance-format#9000. ## What this changes The two checks are separate now: a non-2D array is still rejected with the same message, and the count is compared only when the caller set `num_partitions`, with a message that names both numbers. The Rust core already resolves the partition count from the centroids when `num_partitions` is absent. The `ivf_centroids` docstring said `num_partitions x dimension`, which no longer describes how the count is derived; it now says the row count determines the number of partitions. ## Test plan `test_pre_populated_ivf_centroids` gets two cases. The first supplies a 7 cluster array with `target_partition_size=250` and no `num_partitions`, then asserts the built index has 7 partitions. 7 tells it apart from the 5 partition index built earlier in the same test and from the 4 that `target_partition_size` would have produced. The second passes `num_partitions=4` against the same array and asserts the new mismatch message. Without the fix the first case raises `ValueError: Ivf centroids must be 2D array: (clusters, dim), got (7, 128)`. - `uv run pytest python/tests/test_vector_index.py` 92 passed, 11 skipped, 1 failed - The failure is `test_create_index_progress_callback_error_before_completion_propagates`, which fails the same way with upstream's `dataset.py` on this machine, so it is not from this change. - `uv run make lint` clean --------- Co-authored-by: Xuanwo <github@xuanwo.io>
…-format#9254) The full-zip decoder summed the definition levels of every non-list layer, including layers outside the first list, while the writer counts only the layers beneath it. When a nullable layer sits above a list -- for example struct<x: list<int32>> with both the struct and the list nullable -- the decoder's threshold lands above the level written for a null list, so it hands a leaf value to items that were written as invisible and every later value shifts by one. Depending on the leaf type this silently corrupts data, trips the structural validity check, or panics while slicing the page buffer. The bytes on disk are correct, so this is a read-path fix: existing files decode correctly once the reader agrees with the writer, with no rewrite. Three other decoders (mini-block, complex-all-null and constant) already had the rule right. Move it into a single shared `max_visible_level` in repdef and call it from the writer and all four decoders so the two sides cannot drift again. Also bounds check the fixed full-zip item walk. It is driven by the page's own control words, so a malformed page could read a truncated control word, advance past the end of the buffer, or exhaust the buffer queue -- each of which aborted the process instead of returning a corrupt-file error. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…rmat#9006) ## Problem `TryFrom<PbIvf> for IvfModel` had two ways to abort on a damaged index file. It asserted that the offsets and lengths vectors matched in length, and `assert_eq!` is compiled into release builds. The v1 centroid branch derived the centroid dimension by dividing the value count by the number of partition lengths, which divides by zero when a file carries centroids and no lengths; integer division by zero cannot be compiled out at all. Fixes lance-format#9005. ## What this changes Return `corrupt_file` from both, with the counts. `rust/AGENTS.md` maps format and integrity failures to that variant and reserves `assert!` for conditions that prevent data corruption. These are read-side guards. Nothing in Lance writes either shape: `add_partition` grows both vectors together and the serializer copies both, so the trigger is damaged or foreign input. ## Test plan `test_ivf_model_rejects_mismatched_proto_lengths` parses a proto with 2 offsets and 3 lengths; with the `assert_eq!` restored it fails on the assertion instead. `test_ivf_model_rejects_v1_centroids_without_lengths` parses v1 centroids with no lengths; without the guard it fails with `attempt to divide by zero`. `test_ivf_model_derives_offsets_from_lengths` pins the normal shape the mismatch check has to leave alone, where offsets are empty and derived by scanning the lengths. - `cargo test -p lance-index --lib` 1233 passed, 3 ignored - `cargo clippy --all --tests --benches -- -D warnings` clean - `cargo fmt --all --check` clean ## Not in this change `IvfModel::find_partitions` still does `self.centroids.clone().unwrap()` (storage.rs:113), and `row_range` and `centroid` index by partition without a bound. Those are post-parse, in different functions, and worth a separate look. Co-authored-by: Xuanwo <github@xuanwo.io>
…#9064) ## What this adds A stable-partition rewrite reads source fragments in scan order and distributes their live rows across an ordered list of destination fragments. Row order is preserved within each destination, but the destination cannot be inferred from source row order as it can for ordered compaction. This PR implements the mapping as one immutable Lance file and exposes it through the `MappingReader` interface introduced in lance-format#9106. It does not integrate dataset lineage traversal or change the legacy compaction reader/writer. ## Encoding The row-map file contains one entry for every physical source row, ordered by source fragment and then row offset: ```text stable_partition.lance ├── label: nullable UInt16 │ ├── 0..N-1 → position in the ordered destination list │ └── null → source row was deleted └── global buffer: cumulative counts matrix ``` For example, a label of `2` selects `destinations[2]`; it is not a fragment ID or destination row offset. The writer receives labels for live rows in source scan order and inserts nulls from the source deletion vectors. This preserves one file row per physical source row, including rows already deleted when the rewrite ran. The counts matrix divides the label column into 65,536-row blocks. Its on-disk layout is: ```text 28-byte header ├── magic: "LSPC" ├── version: u32 ├── representation: u32 (0 = dense) ├── num_destinations: u32 ├── rows_per_block: u32 └── total_source_rows: u64 dense grid └── cumulative u32 count for each (block, destination), block-major ``` The Lance schema metadata key `lance:stable_partition:counts_buffer_index` identifies this global buffer. ## Address translation For source row `(fragment_id, row_offset)`, the reader first converts it to its position in the concatenated source layout. A null label returns a deleted row. Otherwise: ```text destination fragment = destinations[label] destination offset = count(label before this block) + rank(label earlier in this block) ``` Because each destination is written in source scan order, this reconstructs its physical destination offset without storing a complete source-to-destination address table. Reordering rows within a destination is outside this encoding and requires a different mapping type. Opening the mapping reads and validates the counts metadata without reading labels. Point and batch translations lazily read only the touched label blocks; batch requests sweep each touched block once. ## Validation The reader validates the label schema, counts header and dimensions, monotonic cumulative counts, source and destination row totals, label bounds, and per-block label/count agreement. Validated with the stable-partition row-map and mapping-reader tests, workspace Clippy, and `cargo fmt --all`. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…build (lance-format#9011) ## Problem `execute_uncommitted` takes the name out of the builder before it rejects a duplicate, so a build that fails there leaves `self.name` empty. A caller that holds the builder and retries then falls into default-name generation and commits under `<column>_idx` instead of the name it asked for. The method is `pub` on `&mut self`, so holding it across a retry is a supported shape. Fixes lance-format#9010. ## What this changes Read the name without consuming it. The same `take()` in `execute_multi_segment_fmindex` is unobservable, since its only caller consumes the builder, but it is changed for consistency. One behaviour change worth naming: a second `execute_uncommitted` on a builder whose first call already committed under that name now returns `Index name 'X' already exists` where before it quietly succeeded under `<column>_idx`. Nothing in the repo does that, and silently renaming the caller's index is the bug being fixed, but it is a new error on a path that used to succeed. ## Test plan `test_failed_execute_uncommitted_preserves_name` builds `retry_idx`, then runs `execute_uncommitted` on a held builder against the same name so it fails on the duplicate, and asserts the name survived. It then does the retry a caller would do and asserts the dataset still has exactly one index. Restoring the `take()` fails both assertions independently: the field reads `None`, and with that assertion removed the retry commits a second index, so the last check reports `["retry_idx", "text_idx"]` against `["retry_idx"]`. - `cargo test -p lance --lib` 3381 passed, 3 ignored - `cargo test -p lance --lib index::create::` 43 passed - `cargo clippy --all --tests --benches -- -D warnings` clean - `cargo fmt --all --check` clean Co-authored-by: Xuanwo <github@xuanwo.io>
…nce-format#9292) Follow-up to lance-format#9172. That PR gave the fresh tier cross-column *multi-match* by decomposing it into per-column searches; this one gives it cross-column *predicates*, which cannot be decomposed. ## The problem `plan_search` bound the whole query to one column, so a boolean or boost whose leaves named different fields had nowhere to go and was refused: ```rust // must: [alpha in title, beta in body] FtsQuery::Boolean(BooleanQuery::new(vec![ (Occur::Must, MatchQuery::new("alpha").with_column(Some("title"))), (Occur::Must, MatchQuery::new("beta").with_column(Some("body"))), ])) ``` That is one predicate over both fields — an intersection — so the per-column decomposition lance-format#9172 uses for multi-match would answer a different question. It refused rather than guess, and `cross_column_boolean_is_refused` pinned that. The committed path has no such gap: `CrossColumnCompoundQueryExec` serves the bounded fully-indexed shape and the DataFusion fallback serves the rest. Only the fresh tier refused. ## The clause algebra was never missing The committed compound scorer sums MUST (`RequiredConjunctionScorer`), sums SHOULD into the surviving set (`DisjunctionScore::Sum`), excludes MUST_NOT, and subtracts `negative_boost * negative_score` for a boost. `search_boolean` and `search_boost` on the memtable already did exactly that, over a `HashMap<DocumentKey, f32>`. What was missing was the ability to reach two indexes. `FtsQueryExpr` carried no column on its leaves, the scanner's `FtsQuery` was `{ column, expr }`, and `FtsIndexExec` resolved a single `FtsMemIndex`. ## The change **The tree walk moves out of the index rather than the column moving into it.** `FtsMemIndex` keeps its leaf scorers and grows `search_leaf_bounded`; `combine_compound` holds the clause algebra once, parameterized by a leaf evaluator, and both the single-index walk and the new `search_cross_column` go through it. One implementation, two routings — rather than a second copy that has to agree with the first. `FtsQueryExpr` leaves gain an optional column binding, and `FtsQuery::new` binds unbound leaves to its column, so every existing caller behaves exactly as before. **Leaves from different indexes need one visibility cut.** Each index snapshots its own `{partitions, tail}` view and those can disagree about how far the memtable has advanced, while `filter_by_visibility` runs *after* the combine — so a MUST across columns could drop a row both columns actually contain. `search_leaf_bounded` takes the exec's `max_readable_row` and clamps each leaf before the clauses meet. This is the one part of the change that is not mechanical. **The planner generalizes to a column slice.** Per-column granularity and index settings move into `resolve_column_index_contract` and are required to agree. Dropping the single-column rebinding in `build_source_plan` is what lets the base and SSTable arms use the dataset scanner's own cross-column path unchanged — no new code there at all. Row documents only, matching `validate_row_leaf_granularities`. **One uncovered column sends every queried column through the transient store.** The clauses meet on `row_position`, and positions are only comparable within one store, so a mixed maintained/transient routing would compare across two. Re-indexing a maintained column costs one tokenize pass over a memtable already paying for the missing one; the alternative buys that pass and costs the invariant. **`CompoundQueryExec` and `CrossColumnCompoundQueryExec` gain `with_limit`.** Their top-k lives in `FtsSearchParams` rather than an enclosing fetch node and their segment selections are private, so a caller over-fetching to survive a later dedup had no way to raise it without rebuilding the node over a wider domain. ## Ranking A cross-column MUST sums two per-column BM25 scores, each local to its own index and its own tier, so the local-scoring divergence this module documents compounds rather than merely applies. Matching stays exact; only order moves. Same trade lance-format#9172 recorded for multi-match, one step wider. ## Tests `cargo test -p lance --lib mem_wal`: 720 passed. New in `fts_search`: a cross-column MUST returns only the row matching in both columns; a cross-column MUST_NOT drops the row carrying the excluded term in the other field; a cross-column boost demotes rather than drops, and the demoted row's score is strictly lower; a column with no maintained in-memory index still contributes through the transient store; an unbound leaf is refused. `cross_column_boolean_is_refused` is superseded — that restriction is what this lifts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rmat#9262) ## What Adds accessors to `ANNIvfBatchExec` — `query()`, `query_count()`, `dataset()`, `indices()`, `prefilter_source()` — and re-exports the node and `QUERY_INDEX_COL` from `lance::io::exec`. ## Why `ANNIvfSubIndexExec` has had `query()`, `dataset()`, `indices()` and `prefilter_source()` since it was introduced. `ANNIvfBatchExec` landed with every field private and only `try_new`, so a caller that matches it in a physical plan can see it exists but can't read anything out of it. That blocks plan rewriting on the batch path. A downstream rewriter that needs to add a second scan arm to a batch vector search — for example, to cover rows that aren't in the index yet — has to read the query vectors, how many there are, and the dataset and index metadata out of the base plan. The per-query union shape (HNSW, or any `refine_factor`) is already reachable through `ANNIvfSubIndexExec`; the shared-scan fast path is not, so the two shapes can't be handled the same way. Rebuilding the node with a different candidate `k` needs the same fields, since `try_new` takes them all and there's no way to get them back out of an existing node. ## Notes - Purely additive: no behavior change, no signature changes, nothing removed. - `query()`'s doc notes that `key` holds all `query_count` vectors concatenated, which is the one non-obvious thing about reading it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ty but zero-width (lance-format#9018) ## Summary The structural full-zip encoder wrote one all-zero control-word byte per item for pages whose rep/def level buffers are non-empty but have a computed bit width of zero, while the decoder reads such a zero-bit layout as NIL (zero bytes per control word). The mismatch interleaves phantom bytes into the zipped value stream and corrupts every value that follows. This adds a zero-width early return in `build_control_word_iterator`: when `total_width == 0` (i.e. `max_rep == 0 && max_def == 0`), emit `ControlWordIterator::Nilary` regardless of which level buffers are present, so the writer agrees with the decoder's NIL interpretation. ## Root cause `encode_full_zip` (`rust/lance-encoding/src/encodings/logical/primitive.rs`) computes `max_rep`/`max_def` from the per-page sliced level buffers, not the schema. When a page's levels are all zero (e.g. a page holding only valid rows of a nullable column after page splitting), `total_width == 0` and the layout metadata records `bits_rep = 0, bits_def = 0`. The decoder reconstructs `ControlWordParser::new(0, 0)` → `NIL` (`bytes_per_word() == 0`), but the writer selected its branch purely on buffer presence, producing a `Unary8`/`Binary8` writer that emits one `0x00` byte per item (`bytes_per_word() == 1`). In the full-zip layout control words and values are interleaved per item, so the decoder's 0-byte step mis-aligns the value stream by one byte per item. ## Testing - `test_control_words_def_only_all_zero_levels`: unit-level regression for the NIL branch — a non-empty all-zero definition buffer with `max_def == 0` must produce a zero-byte-per-word NIL iterator, and `ControlWordParser::new(0, 0)` must parse zero bytes back (write/read symmetry). - `test_control_words_rep_only_all_zero_levels` / `test_control_words_both_levels_all_zero`: defensive sibling checks for the repetition-only and both-levels branches. - Full-zip round-trip tests and the wire-compatibility suite pass unchanged. Co-authored-by: Xuanwo <github@xuanwo.io>
fixes lance-format#8831. Small sequential reads on `BlobFile` previously submitted a storage range for every call. File consumers reading in small chunks could spend most of their time waiting for object storage. This adds lazy read-ahead to Rust `BlobFile`, shared by Python, Java, and Rust callers. It replaces the Python `BufferedReader` from the earlier revision. The default is 4 MiB. `buffer_size=0` disables read-ahead. Explicit `read_range` and `read_ranges` calls bypass the buffer and leave the cursor unchanged. Seeks within the buffered span reuse it; seeks outside discard it. ```python blob = ds.take_blobs("assets", indices=[0])[0] blob = ds.take_blobs("assets", indices=[0], buffer_size=0)[0] ``` Buffering reduces requests for sequential reads but can fetch unused bytes for header probes and scattered reads. The size is configurable; the default has not yet been validated against S3. ## Testing added coverage for sequential read coalescing, disabled buffering, seeks, buffer resizing, and independent range reads. Python tests also cover reads across buffer boundaries, `TextIOWrapper`, ZIP, read-only destinations, and closed handles. Java tests cover disabled buffering and invalid buffer sizes. cc: @wjones127 --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…rmat#9023) A manifest row's `size` attribute was read with `.ok()`, so a value that is not a number silently became `None`. The sibling `version` field, parsed from the same item seven lines below, reports the same failure with a descriptive error: ```rust let size = item.get("size").and_then(|attr| match attr { AttributeValue::N(size) => size.parse().ok(), // silently None _ => None, }); ... let version = version.parse().map_err(|e| Error::invalid_input(format!( "dynamodb error: could not parse the version number returned {}, error: {}", version, e)))?; ``` Losing the size is not free either — `read_manifest` falls back to a `HEAD` request for the length, so a corrupt row costs an extra round trip on every read and gives no signal why. Read the attribute in one place: an absent value stays `None` for rows written before the column existed, and a present one that is not a number is reported. Co-authored-by: Xuanwo <github@xuanwo.io>
…rmat#9025) ## Problem The code-width check in `RabitQuantizationStorage::try_from_batch` is gated on the expected width being positive, so metadata that carries no code dimension (`code_dim` absent and no rotation matrix to recover it from) makes the check compare against zero and skip itself, and a code column of any width loads. Fixes lance-format#9024. ## What this changes Reject `rotated_dim() == 0` before the width check, as `CorruptFile`, and drop the `> 0` gate so the width comparison always runs. No in-tree writer produces that metadata: since lance-format#6024 the writer always sets `code_dim`, and before that it always stored the rotation matrix that `parse_buffer` recovers the dimension from. So this is corrupt-metadata handling rather than a fix for a state a lance version can write, which is also why the `> 0` gate was not protecting a real case: it arrived in lance-format#7021, after every writer already emitted `code_dim`. ## Test plan `test_degenerate_metadata_rejects_any_code_width` deserializes `{"num_bits":1,"packed":true}`, checks `rotated_dim()` is 0, and asserts that an 8-byte code column is rejected with `CorruptFile`. Restoring the `> 0` gate makes it fail: the call returns a storage with `code_dim: 0` alongside an 8-byte code column. `test_metadata_without_code_dim_loads_from_the_rotation_matrix` covers the shape the rejection must not touch, which is the pre-lance-format#6024 one: no `code_dim`, dimension recovered from the rotation matrix that `parse_buffer` installs. - `cargo test -p lance-index --lib vector::bq` 232 passed - `cargo clippy --all --tests --benches -- -D warnings` clean - `cargo fmt --all --check` clean Co-authored-by: Xuanwo <github@xuanwo.io>
…ance-format#9170) Depends on lance-format#9153. This PR's diff currently contains that commit; it reduces to this change alone once lance-format#9153 merges. ## What this does Exact phrases (slop 0) confirm a candidate by aligning every clause's position list. Term frequency equals the number of positions, so a stopword carries the most expensive list — and most candidates die on the rarest pair long before that list matters. Order the clauses by this document's frequency and decode the rarest first, using each of its positions as a candidate base. The remaining clauses are decoded lazily, rarest first, and the scan stops at the first miss, so when the rare pair never aligns the stopword's position block is never unpacked. Above 16 clauses the order and cursors move to the heap instead of inline arrays. This also removes the previous lockstep machinery (`absolute_position`, `advance_to_absolute`) and the per-candidate cursor vector plus sort on the non-bulk path; both paths now share `exact_phrase_positions_match`. The anchor choice cannot change the result: an anchor enumerates all of its positions and each base is verified independently against every clause, so a complete match is always found. ## Measurements Incremental: layer 1 (the phrase position-decode skip, lance-format#9153) -> this commit, same machine. Wikipedia SBG A1 TOP_10, best of 10, one pinned core, all LANCE_HACK_*/LANCE_DIAG_* overrides unset; median of three repetitions: split n before after delta phrase (all) 300 3171us 2831us -10.7% phrase, 2 tokens 198 1337us 1329us -0.6% phrase, 3 tokens 83 4267us 3919us -8.1% phrase, >3 tokens 19 17494us 13737us -21.5% "the book of life" 69922us 45372us -35.1% "the news journal" 15825us 12385us -21.7% "to be or not to be" 217593us 177949us -18.2% Two-token phrases have no stopword to skip, so they are flat; the win grows with the number of clauses. Result counts are bit-identical (943/943 queries). Non-phrase shapes never reach the changed functions and stay within run-to-run noise (union +1.7%, intersection +0.5%, intersection_union -5.7% at n=40). ## Validation - `lance-index` phrase tests pass, including 8 added by this change. - `cargo fmt --all --check` and `cargo clippy -p lance-index --tests -- -D warnings` clean. ## Stack Stack, 2/2: lance-format#9153 -> **this PR**. Base: lance-format#9153. Merge in stack order; retarget/restack the remaining PRs after their parent merges. --------- Co-authored-by: arrowbowang <arrowbowang@tencent.com>
…er (lance-format#9275) Decode `DataFragment.inline_row_ids` as `Bytes` and store it in `InlineRowIds` as such. With the manifest decoded straight from the fetched chunks (previous commit), prost's `copy_to_bytes` now slices the chunk instead of copying into a fresh `Vec<u8>`, which for a compacted stable-row-id table is ~98% of the manifest. Converting back to protobuf for a write clones the `Bytes` handle instead of `to_vec()`. Benchmark (a large stable-row-id table, 945 MiB manifest, 927 MiB of inline row ids, `--profile bench`): decode local manifest, warm: 352-378 ms -> 195-259 ms page faults, 3 decodes: 254k -> 30k DatasetBuilder::load() on S3: 1.9-2.15 s -> 1.34 s peak RSS of open: 993 MB -> 1014 MB (unchanged) Cumulative against the pre-series baseline: decode 826 ms -> ~200 ms, open 2.4-2.7 s -> 1.34 s, peak RSS 1885 MB -> 1014 MB. With `GLIBC_TUNABLES=glibc.malloc.hugetlb=1` decode is 164-189 ms and open 0.98 s.
`import lance` eagerly imported `lance.namespace`, which pulls in the
generated `lance_namespace` REST client with every API module. That was
1.24 s of a 3.3 s import on this machine, paid by every process even
though most never touch a namespace.
Resolve the `LanceNamespace` / `DescribeTableRequest` re-exports through
a module `__getattr__` on first access and import `DescribeTableRequest`
inside `dataset()` only when a `namespace_client` is given. Public names
and `__all__` are unchanged.
Benchmark (`python -c "import lance"`, warm file cache, 3 runs):
before: 3225-3461 ms
after: 880-951 ms
Lance v1.0 through v11.0 decode the inline transaction (manifest field 21) while opening a version, and fail if it contains an operation they don't know. So any new operation type written inline makes that version unopenable for older readers. This PR moves the inline transaction to a new field 23, which those readers ignore. Field 21 is still read, so existing datasets are unaffected, but it is never written again. It also makes transaction decoding total, so this cannot recur: a transaction whose operation is unrecognized now decodes as `Operation::Unknown` instead of erroring. Conflict resolution treats `Unknown` as conflicting with everything, and it cannot be re-encoded or committed. Fixes lance-format#9454 This PR also documents the new field and the handling of unknown operations in the format spec (`docs/src/format/table/transaction.md`). ## Breaking changes For Rust users of `lance-table`/`lance`: - `Operation` has a new `Unknown` variant, so exhaustive matches need an arm for it. The variant is `#[non_exhaustive]`, so it can only be matched (`Operation::Unknown { .. }`), not constructed, outside `lance-table`. - `From<&Transaction>` for `pb::Transaction` and `format::Transaction` are now `TryFrom`, failing on `Operation::Unknown`. ## Not included `index_section` has the same eager-decode hazard; it will be handled separately in the same way. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… first (lance-format#9728) ## Summary Make `remap_column_index` visit every segment of the named index, rather than stopping after the first one. After compaction or reclustering, an index may still store old row addresses. FRI translates those addresses during queries. Remapping rewrites the addresses inside the index so it no longer needs those mappings; it does not rewrite the table's data. ## What was broken? A logical index can contain several physical segments: | Segment of `price_idx` | State | | --- | --- | | S1 | Already uses current row addresses | | S2 | Still needs address remapping | | S3 | Still needs address remapping | Previously, `remap_column_index("price_idx")` selected only S1. If S1 was already caught up, the call returned without reaching S2 or S3. Repeating the call could keep selecting the same unchanged S1. The remaining segments continued to depend on FRI history, preventing trim from releasing their mappings even though the caller had requested a remap. ## What changes? One call now visits all segments selected for that index name, in manifest order: - **Already caught up:** leave the segment unchanged and continue. - **Can be remapped:** rewrite and commit that segment, then continue. - **Blocked or unsupported by the existing remap planner:** skip that segment and continue. Keep the history it still needs. - **I/O or corruption error:** stop and return the error. Each segment is planned against the dataset snapshot current at its turn. On tagged FRI tables, segments are selected from stored metadata, including segments hidden from the query listing. The legacy v0 path keeps its existing query-listing lookup but also visits all matching segments. **Each remapped segment is a separate commit.** This is not an all-or-nothing operation over the entire index: if a later segment fails, earlier successful commits remain. ## Tests and scope New tagged-FRI regression tests verify that: - A later segment is remapped even when the first is already caught up. - One call remaps both segments that need it; a second call makes no further commits. - A blocked segment does not prevent another segment from being remapped. - Remapped files contain destination addresses, queries remain correct, and a subsequent trim releases only history no segment still needs. This fixes an explicitly requested remap. It does not schedule remapping automatically, add it to `optimize_indices`, or run cleanup as part of the call. History trimming remains a separate operation. No persisted-format or public API signature changes. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…#9735) Make `Scanner::project_with_schema` public, and fix the projection path underneath so that exposing it is safe. ## Public API | item | change | |---|---| | `Scanner::project_with_schema` | `pub(crate)` to `pub`. No signature change. | | `ProjectionPlan::from_schema` | Behaviour. A field the dataset lacks, or a type mismatch, is now an error instead of a silently narrower or wrong projection. | ## Why `project_with_schema` should be public It projects by a possibly partial schema: a nested field narrowed to a subset of its children. Expression-based projection cannot express that, since `project` flattens `metadata.location.x` into a top-level column of that name. Only the mem-wal scanner can reach it today, so an external caller must rebuild the nested shape by hand. ## Companion fix: validate the schema where it is accepted `from_schema` is where a caller's schema becomes a set of field ids, but it checks only top-level names. Everything below that is taken on trust, and field ids are dataset-scoped identities, so trusting them only works when the schema came from the same dataset. Two shapes get through: - a nested child the base lacks resolves to nothing, so a caller asking for two children receives one; - a schema numbered independently carries ids that can land on unrelated base fields, so the projection reads the wrong column. Both trip a `debug_assert` in debug builds and pass silently in release. The fix resolves the requested fields through `Schema::project_by_schema` with `OnMissing::Error` and `OnTypeMismatch::Error`, matching by name, rewriting ids to the base's own, and rejecting what the dataset does not have. The take path already validates this way before calling here, and no in-crate caller depends on the lax behaviour. Name resolution rather than an id check is deliberate: a caller holding an Arrow-derived schema should not have to know how Lance numbers fields internally. It also catches type mismatches, which an id check cannot. ## Tests Each fails without the change it covers. - `projection.rs`: a nested child absent from the base is reported rather than dropped. - `projection.rs`: an independently numbered schema resolves by name rather than by colliding id. - `scanner.rs`: a partial nested projection keeps its shape, carries the legacy row-id and row-address flags, preserves `BlobHandling`, and matches `take_rows`. - Doc-test: the scanner test is in-crate and so blind to visibility, so a doc-test covers it, failing to build at `pub(crate)`. ## Related validation gaps found while writing this None is a live bug, and none is addressed here. - **`Projection::union_schema`** documents that fields are added "only if they exist in the base schema" and that it "will panic" on a non-negative id the base lacks. It does neither: such ids are inserted unchecked, and the `debug_assert` meant to catch them sits in the `id < 0` branch it cannot reach. All four callers pass base-derived ids, so nothing breaks today; the contract and the dead assertion mislead. - **`ProjectionRequest::into_projection_plan`** pre-validates only when the schema has no system columns, since `project_by_schema` cannot handle `_rowid`. This PR closes that branch, `from_schema` now validating after system columns are separated out, which also makes the outer special-casing redundant. - **`Projection::union_arrow_schema`** passes `OnTypeMismatch::TakeSelf`, silently coercing a field whose type disagrees with the dataset, where the path hardened here errors. ## PR History - lance-format#5134 proposed adding such a method. Closed as stale. - lance-format#9287 re-raised it and was approved but conflicted with.. - lance-format#8970 which also added the same API, as `project_with_schema` for the mem-wal scanner, but with `pub(crate)` visibility. This PR opens visibility and adds validation. --- **Tracking:** The visibility change is ported in part from rerun-io#72; the projection fix was found while preparing it. --------- Co-authored-by: Clement Rey <cr.rey.clement@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…9757) Java releases currently publish only to Maven Central. Add one step after Central publication to unpack the same signed bundle and upload its files to R2 with the runner's AWS CLI. Stable, beta, and RC releases are copied to `https://maven.lance.org`; dry runs do not upload. This is the transitional dual publishing described in lance-format#9736. The repository supports Maven/Gradle dependencies with explicit versions. This step copies the bundle's POM, JARs, signatures, and checksums; it does not generate version-discovery metadata. An R2 failure fails the existing publish job, so re-running that job also repeats Central publication. Configuration uses GitHub Actions secrets `R2_ACCESS_KEY_ID` and `R2_SECRET_ACCESS_KEY`, and variables `R2_ENDPOINT` and `R2_BUCKET`. These are configured for the `lance-maven` bucket behind `maven.lance.org`. Manual validation ran the exact upload step against an isolated prefix in the production R2 bucket using all 24 released `13.1.0-beta.2` files, including the 208 MB main JAR. Maven resolved that explicit version through the public domain from an empty cache, without version metadata or Central fallback for Lance; the downloaded JAR's SHA-256 matched the released artifact. Test objects were removed. Validation used AWS CLI 2.36.49 and a short-lived bucket-scoped credential, rather than reading back the CI secrets.
…at#9293) Blob v2 compaction currently copies Packed and Dedicated payloads because their descriptors address sidecars through the containing data file. Managed descriptors carry a relative URI, byte range, and optional registered base ID, allowing compaction to preserve payload objects when replacing data files. New payload objects use `_blobs/<uuid>.blob`. Existing Packed and Dedicated objects can be adopted at their original paths without copying bytes, including when their original data files are later removed. Cleanup follows references from protected table versions. The default table base stays implicit, so moving the complete dataset does not retain its old URI. Shallow clone preserves the source base through existing data-file metadata. Deep clone copies local objects at the same relative paths without rewriting their descriptors; references to registered bases are copied and rebound into the destination. Managed uses the existing five descriptor fields in both file formats 2.2 and 2.3. Kind 4 uses the implicit table base; kind 5 carries an exact registered base ID, preserving IDs such as zero in existing shallow clones without changing descriptor nullability. Writer-side `base_id=None` selects the implicit base. Explicit IDs must already exist; writes do not register bases automatically. New Blob data-file commits publish paired reader/writer table capability bits. The capability survives restore; metadata-only updates to an old table do not activate it. Existing kinds retain their read semantics. This adds no user configuration or repack API. Compatibility validation uses a checked-in v11.0.0 Packed/Dedicated fixture. The released v11.0.0 client refuses normal opens of flagged snapshots. A real cached-handle experiment also confirms the boundary: after new-client compaction and cleanup remove the original data files, old-client GC with `delete_unverified=True` can still delete adopted sidecars, while leaving `_blobs` objects alone. Before activation, upgrade all maintenance clients to a Managed-aware version or stop their maintenance tasks. All maintenance after activation must use a Managed-aware client; the flag blocks normal opens of flagged snapshots but cannot revoke pre-existing handles or retrofit old binaries' historical maintenance paths. Cleanup still rescans descriptors shared across metadata-only versions. Scan deduplication is deferred to lance-format#9435 for separate design review. Regression coverage moves complete tables containing both new Managed objects and adopted released sidecars, removes the original location, then scans, reads, and runs cleanup at the new location. Clone coverage checks source retention after compaction, registered-base rebinding, and byte-for-byte preservation of local data files during deep clone. Prepared writes reject unknown bases, including nested Blob columns. The format-spec proposal remains separate, as required by the repository's spec/implementation workflow. Related: lance-format#8899 addresses the compaction problem through a Blob Reuse Index; lance-format#9193 documents the existing Blob v2 contract. ## Benchmark At the benchmarked revision, Managed kept total write amplification close to 1x through ten compactions, with approximately unchanged full-scan and random-read performance on the tested workloads. Benchmarked revision: `f7dec16f659f758d2b7b302513501975767fda78`; historical baseline: `400408ffb7ee6b7bbf7c240962b5a94fb5fef879`. Runs used one dedicated EC2 **c7i.24xlarge** (96 vCPUs, 192 GiB RAM) and same-region S3 in **us-east-2**, Rust 1.97.0, `release-with-debug` with thin LTO, and 48 CPU/runtime threads. Both builds used identical dependency locks and CPU flags (`haswell`, `+avx2,+fma,+f16c`). Data uses deterministic high-entropy payloads in format 2.2. An explicit 8 KiB inline threshold and 512 KiB Dedicated threshold select the requested layouts; the production Dedicated default is 4 MiB. These measurements apply only to the pinned revision above. Subsequent changes include upstream BlobFile buffering/read changes, dependency updates, and portable Managed base resolution. The numbers have not been revalidated for the current head and are retained as historical evidence, not a current-head performance claim. Cleanup scan deduplication remains in draft follow-up lance-format#9435 and is outside these measurements. ### Write amplification For each payload size, append **10,000 rows** at a time to **1,000,000 rows**, compacting the accumulated table into one fragment after every additional **100,000 rows**. No garbage collection runs. Total written bytes include appends, all ten compactions, descriptors and metadata; submitted write bytes were cross-checked against the retained S3 inventory. GB below is decimal. | Payload/layout | Logical input | Total written by this PR | Write amplification | |---|---:|---:|---:| | 100 KiB packed | 102.400 GB | 102.672 GB | 1.002653x | | 1 MiB Dedicated | 1,048.576 GB | 1,048.948 GB | 1.000355x | Retained payload bytes equal logical input bytes for both cases: compaction did not rewrite the payloads. The historical full ingestion/compaction workflow was not measured, so these are candidate measurements, not a measured old/new write-amplification ratio. ### Full-column scans Read the entire blob column, including every payload and its checksum. Use 16 MiB logical batches, batch/fragment readahead 16, fresh processes, one excluded warmup per version, and **three alternating old/new pairs**, with no concurrent writes or other scans. Both versions read identical 1M-row contents. The candidate uses its actual final compacted tables. Baseline final tables were generated directly with the old compaction writer's batch/row-group settings; a small preflight checked payload sizes, layout kinds and full contents against real old compaction output. This compares final-table reads and does not replay the old write workflow. Packed has 96 old versus 100 new payload objects; Dedicated has 1M in each version. | Payload/layout | Old median GB/s (range) | New median GB/s (range) | Median paired throughput change | |---|---:|---:|---:| | 100 KiB packed | 0.836 (0.832–0.875) | 0.834 (0.826–0.841) | -1.22% | | 1 MiB Dedicated | 0.620 (0.580–0.644) | 0.626 (0.597–0.643) | +0.88% | ### Random access: take 10 complete blobs Reuse those same final tables. Each request selects ten distinct random row indices, calls `take_blobs_by_indices`, and concurrently reads all ten `BlobFile` payloads. Request batches are serial. Timing includes descriptor lookup, payload reads and checksum hashing; dataset open and expected-content generation are excluded. Each process reuses its dataset session, excludes ten warmup requests, then measures 100 requests. **Five alternating pairs** give **500 measured requests per version/layout**, with matching random indices within each pair. Every returned payload and its row order were validated. This measures reused sessions; payload caches were not explicitly flushed. | Payload/layout | Old P50 / P95 latency | New P50 / P95 latency | Median paired P50 latency change | |---|---:|---:|---:| | 100 KiB packed | 157.03 / 220.27 ms | 157.81 / 212.53 ms | +0.05% | | 1 MiB Dedicated | 149.97 / 193.99 ms | 149.59 / 187.05 ms | +0.61% | Latency percentiles use nearest-rank over the 500 requests. Paired changes use the median of within-pair new/old ratios (round P50s for random reads), not ratios of the aggregate table medians. Lower latency is better. These small differences do not establish a sustained read improvement or regression; results describe this configuration rather than a hardware throughput ceiling. Raw commands, checksums, timings, binary hashes and exact dependency locks are retained in the private benchmark project.
…rmat#9750) A DataFusion-planned `ORDER BY ... LIMIT` over a Lance read (e.g. SQL through `LanceTableProvider`) reads every projected column for every matching row, then the top-k discards most of them. `TableProvider::scan` carries no ordering, so nothing narrows the read. `TopKLateMaterialization` is a physical optimizer rule that narrows the read under a top-k to the sort columns plus `_rowid` and takes the remaining columns for the surviving rows, as `Scanner::order_by` already does: ``` Projection -> Take(other columns) -> [SortPreservingMerge] -> Sort(TopK) -> ... -> FilteredReadExec(sort keys + _rowid) ``` - It walks through column-selecting projections, non-hash repartitions, `CoalescePartitionsExec` and `CooperativeExec`. A custom node, such as one a `TableProvider` wraps around the scan, is walked only when the caller certifies it with `with_pass_through`, since a row-preserving node may still change values. - The narrowed read keeps the original's precomputed plan and session-planned filters. - The scanner's own late materialization behind a filter (a `TakeExec` or row-stream `FilteredReadExec`) is dropped; the take above the sort fetches those columns. - `TakeExec` emits the dataset's schema metadata, so a wrapper that rewrites metadata is applied again above the take to keep the plan's schema. - `with_read_eligibility` lets a caller decline reads a take cannot serve. It is exported but not added to `get_physical_optimizer()`, since the scanner plans its own order-by. ENT-2777
…mat#9699) `fast_search()` changed the requested projection, causing FTS and vector queries selecting only `id` to return an extra `_rowid` column. Remove the row-ID projection mutation from `fast_search()`. Search nodes already produce the row IDs needed internally, and the take step preserves them. Explicit `_rowid` projections and `with_row_id()` continue to return row IDs. This also preserves deleted-row validation and makes scalar-query plans independent of whether `project()` runs before or after `fast_search()`. Regression coverage includes FTS and vector search, default and explicit projections, requested row IDs, and both row ID modes over two fragments. It checks result equality with normal search and recall 1.0, rejects deleted-row scans without requested row IDs, and compares scalar-query plans and results across both call orders. Validation: - `cargo test -p lance fast_search --lib` — 38 passed. - `cargo fmt --all` — passed. - `cargo clippy --all --tests --benches -- -D warnings` — passed. Fixes lance-format#9696 <!-- lance-gatekeeper-fix:v1 agent=1e7c4cb11ebe181331d237e64965edf8 generation=1 --> --------- Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com>
…pening it (lance-format#9712) A schema change on a MemWAL-backed table — a column added, dropped, renamed at any depth, or relaxed to nullable — lands on the base table only. A live `ShardWriter` keeps the schema it was opened with, so it rejects every write shaped to the new one, and the only way forward is to reopen it: re-read the WAL tail since the last flush, re-flush what fills during replay, and rebuild every in-memory index. The shard takes no writes while that happens. `ShardWriter::evolve_schema` moves the writer instead. It seals the active memtable under the schema it was written with and starts the next one under the new schema, so the sealed one flushes as a generation of its own schema with the index configs it was built with. An empty memtable is replaced rather than sealed. Nothing is replayed and no index is rebuilt. `ShardWriter::evolve_to(&Dataset)` derives both halves from a dataset. The writer side of this change is small. Most of the diff is the read side: the memtable sealed at a schema change stays in memory until it flushes, still holding its rows under the old column names, so every read path now has to resolve such a memtable. ## Reading across the change A sealed generation already resolves to the table's schema by field id. An in-memory memtable did not — pairing one with a schema it was not created from was documented as outside the contract. Now a memtable whose schema no longer matches the table resolves the same way, on every read path: | arm | a memtable created before the change | | --- | --- | | scan | projected and filtered under its stored names; a predicate it cannot answer runs above the resolution, still after the dedup | | point lookup | resolved through the plan path; the probe fast paths are skipped while any memtable predates the table's schema | | vector search | searched on the column's stored name; with a deferred predicate it ranks every row exactly before the filter | | full-text search | queried and looked up under stored names, the index contract included | A memtable that matches the table — every memtable in the steady state — keeps exactly today's path, behind a positional check. ## Writing safely across it A put is validated and conformed under the writer lock, against the schema of the memtable it lands in, so a write racing a schema change cannot be checked against one schema and inserted under another. The logical, storage and blob schemas, the primary key and the index configs move together as one `WriterSchema`. ## Recovery A WAL entry now records the generation its rows belong to, beside the writer epoch it already carried. Replay reads the boundary instead of measuring it, which is what lets a generation end at a schema change rather than at a row count. A WAL-only writer holds no memtable and records none; replay still sizes those entries itself. Where a recorded generation will not fit the memtable replaying it, replay refuses before publishing anything, so the shard is untouched and reopens under the configuration it was written with. This is a format addition — an older reader ignores the key and falls back to measuring — and `docs/src/format/table/mem_wal.md` documents it. Two bugs surfaced on the way and are fixed here: replay indexed only its final memtable, so any generation it sealed mid-replay was published with no primary-key sidecar and its rows could not be looked up by key; and the writer did not own the entry's reserved metadata keys, so a caller's schema metadata could name a generation the entry did not have. ## Testing `mem_wal`: 809 passing, `fmt` and `clippy -D warnings` clean. New coverage for each read arm across a rename, an add and a drop; for a nested path, which only resolves because the field-id map pairs full paths rather than top-level names; for the recorded generation, including a boundary nothing else can measure; and for the capacity refusal, asserting the manifest is unchanged by it. --------- Co-authored-by: Daniel Rammer <hamersaw@protonmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: XYZhan <xuanyu.zhan@lancedb.com>
## Problem
`MultiMatchQuery`'s custom serde wrote only `{query, columns, boost}`,
taken from the first leaf, and decoded every leaf as a default
`MatchQuery`. Each leaf's terms, operator, fuzziness, `max_expansions`,
`prefix_length` and document granularity were lost over JSON, so an
`AND` multi-match came back as `OR`.
Anything that ships an `FtsQuery` as JSON hits this. In LanceDB that
includes the remote client's structured FTS body
(`rust/lancedb/src/remote/table.rs`), so `MultiMatchQuery(operator=AND)`
from Python is evaluated as OR server-side.
## Fix
- **Serialize:** every leaf goes out as a full `MatchQuery` under
`match_queries`, the shape the JSON parser
(`MultiMatchQuery::from_json`) and the namespace spec already use.
- **Deserialize:** `match_queries` is read first.
- **Compatibility:**
- When every leaf names a column, the legacy `query`/`columns`/`boost`
keys are still written. A reader that predates `match_queries` ignores
the new key and degrades exactly as it does today, so a rolling upgrade
doesn't break plain multi-matches.
- When a leaf has no column, only `match_queries` is written. An old
reader then fails with `missing field "query"`; the old writer panicked
on the column `unwrap()`.
- The legacy shape still decodes unchanged.
No other `FtsQuery` variant's serde loses fields. Boost and Boolean only
lost data when nesting a multi-match, which the nested round-trip test
covers.
## Tests
- `test_multi_match_query_serde_round_trip`: per-leaf AND, fuzziness,
max_expansions, prefix_length, boost, granularity, and different terms
per leaf.
- `test_multi_match_query_serde_nested_round_trip`: a Boost wrapping a
Boolean that contains a MultiMatch.
- `test_multi_match_query_legacy_serde`: the legacy shape decodes as
before; new output still carries the legacy keys.
All three fail without the fix. `cargo test -p lance-index --lib
scalar::inverted::` passes (912 tests), and clippy for `lance-index` is
clean.
## Downstream
Two LanceDB tests encode the old behaviour and will need updating on the
bump:
- `python/python/tests/test_fts.py:1267` asserts the exact multi-match
JSON string.
- `rust/lancedb/src/table/datafusion/udtf/fts.rs:1488` expects one row
from leaves with different terms; it only passed because the second
leaf's terms were dropped.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…format#9435) Cleanup repeatedly scans Managed Blob descriptors when table versions share the same fragment view. This change records successful scans separately for retained and expired versions and adds discovered paths directly to the existing cleanup sets. Each view is scanned at most twice per cleanup, without retaining a separate copy of its full reference set. The scan identity includes fragment metadata (including deletions and overlays), projected fields and base bindings. It uses the implicit and registered base semantics from lance-format#9293. Branch retention shares the same inspection state; failed reads are never recorded as successful scans. A missing expired descriptor can be skipped, while a missing retained descriptor still stops cleanup before deletion. ## Benchmark Compared main `84d1681e07af565837f2d97e72ff1b8b040cd15c` with this PR at `9738f9ef767307f3e4695695b8d3915189d06f41`. Each case contains 100,000 real Managed objects. Metadata-only versions share two descriptor files; compaction cases replay cumulative appends followed by 8 or 32 compactions (24 or 96 table versions). The measured workflow is `dataset.cleanup(policy).with_max_candidate_files(0).explain()`, retaining the latest version and selecting older versions for cleanup. This measures inspection and file listing, without deleting files, so both versions use identical inputs. The 64-byte Dedicated payloads isolate descriptor work; cleanup does not read payload contents. Every latest payload was separately verified. One Apple M4 Max / 128 GiB / local APFS host, Rust 1.98.1, identical dependency lock and `release-with-debug` builds, eight CPU/I/O/runtime threads. Each version has an excluded warmup followed by five alternating pairs in fresh processes, with warm OS caches and no concurrent builds or dataset writes. Values below are medians; RSS includes dataset opening but excludes generation and payload validation. | Workload | Main inspection time | PR inspection time | Main peak RSS | PR peak RSS | |---|---:|---:|---:|---:| | 8 metadata-only versions | 1.456 s | 0.735 s | 215.8 MiB | 97.6 MiB | | 128 metadata-only versions | 15.089 s | 0.693 s | 373.6 MiB | 94.6 MiB | | 8 compactions | 1.911 s | 1.037 s | 223.2 MiB | 162.1 MiB | | 32 compactions | 6.046 s | 2.386 s | 272.4 MiB | 194.0 MiB | Median paired elapsed-time reductions are 49.6%, 95.4%, 45.8% and 60.5%, respectively. For 128 metadata-only versions, elapsed ranges are 14.553–15.469 s on main and 0.683–0.717 s on this PR. For 32 compactions they are 5.873–6.224 s and 2.319–2.493 s. These results cover local cleanup inspection, not S3 performance or the final deletion phase. Global path sets and metadata for distinct fragment views still consume memory. All paired cleanup removal statistics matched. Separate execution probes cover actual deletion, tagged and branched retention, dropped Blob columns and missing retained descriptors. Commands, all ranges, raw paired measurements, build hashes and dependency locks are preserved in the private benchmark project; the additional probes are standalone evidence, not new public CI coverage. The September S3 scan benchmarks in lance-format#9293 do not measure this change.
`LanceDataset.get_version()` returns the checked out version with its timestamp and manifest summary. It mirrors `Dataset::version()` in Rust and `Dataset.getVersion()` in Java, and has the same shape as an element of `versions()`. Until now the summary and the timestamp were only reachable through `versions()`, which reads every manifest. `get_version()` reads the manifest that is already loaded and does no I/O.
…rmat#9776) Nullable vector builds do full-column non-null counts before training after lance-format#9134. Sampling wide nullable vectors also falls back to one row per batch when the local storage block is smaller than a vector. - Stop preflight counting once the required training threshold is met; scan to exhaustion when there are not enough vectors. - Read uniform random samples in batches targeting 64 MiB. Bound batch prefetch by `sample_size * vector_byte_width` and I/O parallelism. - Consume each sampling round before refilling NULLs, preserve sampling without replacement, and shuffle the output so training prefixes are not ordered by row ID. Validation: - `cargo test -p lance --lib --profile ci index::vector::`: 364 passed, 1 existing ignored test. - `cargo clippy --all --tests --benches --profile ci -- -D warnings`: passed. - `cargo fmt --all -- --check`: passed. ### Performance Same AWS r8id.4xlarge (16 vCPU, 128 GiB RAM, local NVMe), Rust 1.98.1, repository `release-with-debug` settings. CPU, I/O, and Rayon parallelism were set to 16; the logger was disabled in both builds. Baseline: parent `ef1fb2959`; this PR: `5754c9402`. Full Coyo, revision `95efb82f4914f320e59cf286a06f23b7cfdb2112`: 15,380,795 rows, nullable 2048-dimensional float32 vectors. Each call returned 1,048,576 non-null vectors (4096 partitions x sample rate 256). Median of three fresh-process runs per variant and scenario, with page cache dropped before every process. Variant order was reversed in the second repetition. Cold-cache sampling measures `maybe_sample_training_data` directly. The preparation scenario runs the two training-floor checks (256 and 1,048,576), then samples with the cache state those checks leave behind. Lower is better. | Scenario / metric | Baseline | This PR | Benefit | | --- | ---: | ---: | ---: | | Cold-cache sampling | 39.525 s | 11.705 s | 3.38x speedup | | Preflight counting | 118.966 s | 4.022 s | 29.58x speedup | | Sampling after preflight | 33.139 s | 13.049 s | 2.54x speedup | | Preflight plus sampling | 152.093 s | 17.108 s | 8.89x speedup | Cold-cache sampling read about 8 GiB in both variants. Median peak process RSS increased from 8.08 GiB to 10.17 GiB. The prefetch budget covers vector payloads; the output sample and decoder buffers consume additional memory. These timings exclude normalization, k-means, and the rest of index construction. They do not establish an end-to-end index-build speedup. Related: lance-format#6738 parallelizes null compaction; this PR changes preflight counting and sample reads.
lance-format#9777) Managed Blob capability transitions and cleanup scan reuse need durable regression coverage after lance-format#9293 and lance-format#9435. These tests cover activation on new Blob files, preservation after removing Blob columns, coexistence with base-path and mixed-version flags, and cleanup retention across deletions, projections, tags, and branches. Concurrent scan requests must share one scan per retention class; failed descriptor reads must not publish successful completion or permit unsafe deletion. The cross-version tests use released v11.0.0 and v13.0.0 clients to create Packed/Dedicated data, verify metadata-only updates remain readable, and check that adoption and subsequent restore fence old opens and appends. Historical unflagged reads remain allowed; this does not promise safety for old cached-handle maintenance. Both release checks passed locally through the existing subprocess runner using ordinary CPython 3.14 for the released abi3 wheels. No production behavior changes. The larger Blob v2 specification remains in lance-format#9193.
The feature flag table still treats the Managed Blob bit as unknown, although lance-format#9293 assigned it. Express all bit masks as powers of two and document the Managed Blob capability at `2^14`, the unknown boundary at `2^15`, paired reader/writer bits, and activation and restore behavior, including the old-maintenance limitation. This corrects the flag reference for the implementation already on main. The full Blob v2 specification remains in lance-format#9193; this change does not add descriptor encodings or runtime behavior.
…mat#9644) feat(java): count indexed rows on explicit scalar segments ## Summary - Pin the existing `Dataset.countIndexedRows` method to the named scalar index, and reject filters that cannot be planned against that index. The three-argument signature is unchanged. This corrects the method so it counts with the index it was given, rather than scanning or selecting another index. - Add an overload that opens only the requested physical segment UUIDs. Their current fragment coverage defines the count scope, and an explicit fragment list must match that coverage. A selection that omits a fragment-reuse contributor for that scope is rejected; the error names the missing UUIDs. - Keep validation and count semantics in the Rust core. Deleted rows stay excluded, including on datasets that use stable row IDs. - The three-argument signature is unchanged and the segment overload is additive. This does not change a published method signature, so it should not be labeled as a public API break. ## Test plan - [x] `./mvnw spotless:apply` and `./mvnw spotless:check` in `java/` - [ ] `ScalarIndexTest` covers independent segment counts, A+B versus the full logical index, coverage mismatch, unknown and wrong-index UUIDs, duplicate and empty input, deletions, stable row IDs, and proof that an unselected segment file is not opened - [ ] Local Cargo compile and tests were not run on this machine. Remote CI is the compile and test gate. - [ ] Confirm existing three-argument Java callers still compile --------- Co-authored-by: zhanghaobo@kanzhun.com <zhanghaobo@kanzhun.com>
…mat#9322) ## Summary - preserve numeric literal types in Boolean comparisons such as `flag != (id > 0)` - let DataFusion coerce complex expressions, including mixed numeric arithmetic, without narrowing their literals to an outer or inner column type - retain column-type coercion for literal-only same-type expressions, including shifts and string concatenation ## Root cause The filter resolver coerced every literal inside a right-hand binary expression to the outer left column's type. For a Boolean column, this incorrectly converted literals in nested numeric comparisons to Boolean. Complex expressions now retain their operand types until DataFusion applies type coercion. This also preserves valid filters such as `f = n + 0.5` (`Float64` / `Int64`) and `wide = small + 128` (`Int64` / `Int8`). ## Validation - `cargo fmt --all` - `cargo test --locked --profile ci -p lance-datafusion`: 194 unit tests and 5 doctests passed; 3 existing doctests skipped - `cargo clippy --profile ci --locked --all --tests --benches -- -D warnings` - execution regressions cover Boolean comparisons with Int32/Float32 columns, mixed numeric arithmetic in both operand orders, NULL operands, constant shifts, and string concatenation Fixes lance-format#9319 <!-- lance-gatekeeper-fix:v1 agent=49c00f24983dad193ccadc1df5499a7d generation=1 --> --------- Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Yang Cen <bubble-cal@outlook.com>
# Conflicts: # rust/lance/src/dataset.rs
- ATM, merging schemas where the metadata is different fails / crashes - we need to copy the metadata from the existing field
This reverts commit 71c49d9. # Conflicts: # Cargo.lock # Cargo.toml # java/lance-jni/Cargo.lock # python/Cargo.lock
…adInternalPlan and keep selection when doing empty children
|
ACTION NEEDED The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. For details on the error please inspect the "PR Title Check" action. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix: share filtered-read predicates across fragments
Before
Filtered-read planning deep-cloned logical predicates for each fragment.
Converting the plan into fragment read tasks cloned them again.
Large predicates, such as long
INlists, multiplied memory usage by thenumber of fragments and could cause OOMs.
After
Fragments share immutable logical predicates through
Arc<Expr>, includingwhen creating read tasks. Filter-selection and evaluation behavior remain
unchanged.
Changes
Arcreferences instead of expression trees for each fragment.ScopedFragmentRead.filters share one allocation.
Validation
from approximately 409 MB to 66 MB.