Skip to content

vectors: rerank_oversample on BINARY, queryNodes where, indexed pre-filter (ArcadeDB 26.11.1) - #148

Merged
tae898 merged 3 commits into
mainfrom
docs/26111-vector-changes
Oct 30, 2026
Merged

tae898 merged 3 commits into
mainfrom
docs/26111-vector-changes

Conversation

@tae898

@tae898 tae898 commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator

Brings the vector guide, the rerank_oversample check and their tests in line with three ArcadeDB 26.11.1 engine changes.

Engine change Upstream What changes here
A BINARY index scores nodes on its 1-bit codes and reranks k * rerankOversample candidates (default 10), or the whole efSearch beam when larger; the rerank option works on it ArcadeData/arcadedb#10975 (fixes #10968) create_vector_index(rerank_oversample=...) accepts PRODUCT and BINARY on both backends; NONE, INT8 and no quantization keep the ValueError, as do pq_* and fused_pq without PRODUCT. Guide, docstrings updated
db.index.vector.queryNodes(index, k, vector, {where, filter}) options map ArcadeData/arcadedb#10808 (feat of #10685), first in 26.11.1; 26.10.2 answers expects 3 arguments but got 4 "has no filter" sentence corrected in the guide, example 29 and the examples README; the map is shown once
A selective where on an indexed property takes the property index; broad and unindexed conditions unchanged ArcadeData/arcadedb#10957 (fixes #10921) One paragraph in "Filtered search" (index the property you filter on, no numbers, link to #10921); example 29 now creates NOTUNIQUE on Fact.tenant

Why rerank_oversample is not opened for every quantization: LSMVectorIndex.findNeighbors documents the setting as "Ignored when the index is neither PRODUCT nor BINARY", and LSMVectorIndexMetadata marks it "PRODUCT and BINARY". The metadata parser itself accepts the key on any index, so the stricter Python rule stays for NONE and INT8.

Verification:

  • Tests (python -m pytest, pinned taskset -c 4,5, Python 3.13) ran against a scratch layout of the JVM wheel with the engine jars of upstream 313f23e234 (contains all three changes) and this branch's Python sources overlaid; no wheel was built. 139 passed across test_vector_product_quantization.py, test_vector_sql.py, test_vector.py, test_vector_index_introspection.py and test_docs_vectors_page.py. The same files pass on e353330c5c (has #10808, not #10957 or #10975) and the new queryNodes test and the BINARY tests pass on 26.10.2 jars through their old-engine branches.
  • The BINARY tests fail on the old check (5 failed on origin/main sources over the 313f23e234 jars).
  • Verified from engine source only, not by a test: that a BINARY search reranks k * rerankOversample candidates (a recall difference of a few percent with no deterministic assertion) and that the indexed pre-filter plan is taken (the new test pins the returned rows, which are the same whichever plan runs). Both are covered by upstream's own tests.
  • Version gating follows the repository's patterns (has_pq_rerank probe and _engine_version()), no skips.
  • Example 29 runs against 313f23e234 with its checks passing.
  • black, isort, pre-commit, both bandit commands clean; the changed .py files compile.

🤖 Generated with Claude Code

…ilter (26.11.1)

Three engine changes in ArcadeDB 26.11.1 made statements here wrong.

- #10975 (fix of #10968): a BINARY index scores nodes on its one-bit codes and
  reranks k * rerankOversample candidates (the setting PRODUCT uses), and the
  rerank option of vector.neighbors works on it. create_vector_index refused
  rerank_oversample unless quantization="PRODUCT", on both backends. It now
  accepts PRODUCT and BINARY. Engine source (LSMVectorIndex.findNeighbors,
  "Ignored when the index is neither PRODUCT nor BINARY") says NONE and INT8
  ignore the setting, so those, and every pq_* and fused_pq, keep the
  ValueError.
- #10808 (#10685): db.index.vector.queryNodes takes a fourth argument, a
  {where, filter} options map, since 81c30122fc (first in 26.11.1; 26.10.2
  answers "expects 3 arguments but got 4"). The page, example 29 and the
  examples README said the procedure has no filter.
- #10957 (fix of #10921): a selective where on an indexed property takes the
  property index; unindexed conditions are unchanged. The Filtered search
  section says so and example 29 now indexes the property it filters on.

Tests: BINARY accepts and reads back rerank_oversample (create_vector_index and
a CREATE INDEX statement), still refuses pq_* and fused_pq, search with the
rerank option returns exact distances; NONE, INT8 and None refuse
rerank_oversample; queryNodes {where} returns only matching rows for an indexed
and an unindexed property. Version-dependent branches follow the existing
probe (has_pq_rerank) and _engine_version() patterns, no skips. Ran on the
313f23e234 engine jars, on e353330c5c and on 26.10.2; the BINARY tests fail on
the old check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tae898 and others added 2 commits October 30, 2026 13:09
…the native build too

Native parity on PR #148 failed test_query_nodes_takes_the_where_options_map with
JVMNotRunning: _engine_version() read com.arcadedb.Constants through JPype, which the
native build does not have. The behaviour is shared by both builds, so per the
conftest rule (a test native shares must pass on native) the check now reads
SELECT version() through the test's database. The two older tests that were on the
NA-JVM list only because of that JPype read (test_where_map_and_condition,
test_where_combines_with_filter) come off it.

Checked: the three where tests pass on the native wheel (engine e353330c5c) and the
JVM wheel (same engine), run from a copy of tests/ against the installed wheels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tae898
tae898 merged commit 2d302e8 into main Oct 30, 2026
37 checks passed
@tae898
tae898 deleted the docs/26111-vector-changes branch October 30, 2026 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant