vectors: rerank_oversample on BINARY, queryNodes where, indexed pre-filter (ArcadeDB 26.11.1)

docs/modules/ROOT/pages/how-to/vectors.adoc

@@ -137,7 +137,7 @@ Bind the query vector as a parameter, as the block does, instead of writing the
 
 === openCypher
 
-The Cypher procedure `db.index.vector.queryNodes(index, k, vector)` yields `node` and `score`. Its arguments are in that order, `k` before the vector. A higher `score` is closer. The procedure has no filter, so a `WHERE` after `YIELD` runs on the `k` rows only.
+The Cypher procedure `db.index.vector.queryNodes(index, k, vector)` yields `node` and `score`. Its arguments are in that order, `k` before the vector. A higher `score` is closer. On ArcadeDB 26.11.1 and later a fourth argument, an options map, restricts the search before the top `k` is taken: `where` (a map of property to value, or a condition string) and `filter` (a list of nodes or record ids), for example `CALL db.index.vector.queryNodes('Doc[embedding]', 2, $vec, {where: {name: 'python'}})`. Without the map, and on an engine before 26.11.1, there is no filter, so a `WHERE` after `YIELD` runs on the `k` rows only.
 
 [source,python]
 ----
@@ -195,6 +195,8 @@ vector
 
 In SQL, the `filter` option of `vectorNeighbors` does the same with a list of record ids written in the statement. The `where` option tests a condition on each candidate as the search reaches it (a map of property to value, or a condition string).
 
+On ArcadeDB 26.11.1 and later, a selective `where` on an indexed property uses the property index, in `vectorNeighbors` and in `db.index.vector.queryNodes` alike: the search starts walking the graph, and when the condition keeps few records it lists them through the index and scores them directly. A broad condition keeps walking. Index the property you filter on, for example `CREATE INDEX ON Doc (name) NOTUNIQUE`. A condition on a property without an index still loads candidates to test them, as before. The plan and its measurements are in https://github.com/ArcadeData/arcadedb/issues/10921[ArcadeData/arcadedb#10921].
+
 Filtering the rows afterwards is the simplest way. The search runs first and the filter second, so ask for more neighbours than you need. A filter that drops rows can leave fewer than you asked for.
 
 [source,python]
@@ -247,7 +249,7 @@ INT8 True
 
 Two rules hold for the quantization arguments of the method:
 
-* The `pq_*`, `rerank_oversample` and `fused_pq` arguments need `quantization="PRODUCT"`. The call raises `ValueError` otherwise. The same keys are `METADATA` keys in a statement (`pqSubspaces`, `pqClusters`, `pqCenterGlobally`, `pqTrainingLimit`, `rerankOversample`, `fusedPQ`). With `PRODUCT`, every graph rebuild retrains the codebook and re-encodes every vector (https://github.com/ArcadeData/arcadedb/issues/10847[ArcadeData/arcadedb#10847]), so measure recall and rebuild time on your own data before you choose it.
+* The `pq_*` and `fused_pq` arguments need `quantization="PRODUCT"`. `rerank_oversample` needs `"PRODUCT"` or, on ArcadeDB 26.11.1 and later, `"BINARY"`: a `BINARY` search scores the one-bit codes and re-ranks `k * rerank_oversample` candidates on the stored vectors (default 10), or the whole `efSearch` beam when that is larger (https://github.com/ArcadeData/arcadedb/issues/10968[ArcadeData/arcadedb#10968]). The call raises `ValueError` otherwise. The same keys are `METADATA` keys in a statement (`pqSubspaces`, `pqClusters`, `pqCenterGlobally`, `pqTrainingLimit`, `rerankOversample`, `fusedPQ`). With `PRODUCT`, every graph rebuild retrains the codebook and re-encodes every vector (https://github.com/ArcadeData/arcadedb/issues/10847[ArcadeData/arcadedb#10847]), so measure recall and rebuild time on your own data before you choose it.
 * `encoding="INT8"` is for a `BINARY` property that already holds INT8 bytes. Set `quantization` to `"NONE"` with it. The call raises `ValueError` when both are `INT8`, because the vector would be quantized twice.
 
 == Rebuild the graph

examples/29_hybrid_memory_retrieval.py

@@ -15,8 +15,9 @@
 The pre-filter is the point. Taking the global top-k first and filtering by
 tenant afterwards returns fewer rows than asked, or none, whenever the nearest
 neighbours belong to another tenant (this is what Cypher's
-``db.index.vector.queryNodes(index, k, vector)`` does: it has no filter, so
-``WHERE node.tenant = ...`` runs on its k results; ArcadeData/arcadedb#10685).
+``db.index.vector.queryNodes(index, k, vector)`` does without its fourth
+argument, the ``{where, filter}`` options map of ArcadeDB 26.11.1: a ``WHERE``
+after ``YIELD`` runs on its k results; ArcadeData/arcadedb#10685).
 ``vector.neighbors`` takes the restriction as an option, so the index search
 itself only considers the tenant's vectors:
 
@@ -29,7 +30,9 @@
   has ``where``.
 
 The index types are created with SQL and are always the same: an ``LSM_VECTOR``
-index on ``Fact.embedding`` and a ``FULL_TEXT`` index on ``Fact.text``.
+index on ``Fact.embedding`` and a ``FULL_TEXT`` index on ``Fact.text``, plus
+a ``NOTUNIQUE`` index on ``Fact.tenant``: from 26.11.1 a selective ``where`` on
+an indexed property takes the property index instead of loading candidates.
 ``vector.fuse`` and the ``SEARCH_INDEX`` source are used as documented in the
 ArcadeDB vector-search concept page ("Hybrid Search with vector.fuse").
 
@@ -172,6 +175,7 @@ def build(db) -> dict:
         "CREATE INDEX ON Fact (fid) UNIQUE_HASH",
         "CREATE INDEX ON Entity (eid) UNIQUE_HASH",
         "CREATE INDEX ON Fact (text) FULL_TEXT",
+        "CREATE INDEX ON Fact (tenant) NOTUNIQUE",
         'CREATE INDEX ON Fact (embedding) LSM_VECTOR METADATA {"dimensions": %d, '
         '"similarity": "COSINE"}' % DIM,
     ):

examples/README.md

@@ -430,8 +430,8 @@ Hybrid retrieval for an agent memory in one SQL statement instead of three store
   the same tenant, both fused by reciprocal rank fusion with `vector.fuse`, then graph hops
   (`MENTIONS`, `RELATES`) to related entities and facts, kept inside the tenant
 - shows why: a global top-k followed by a tenant test returns 0 rows when the nearest
-  neighbours belong to another tenant (the behaviour of Cypher `db.index.vector.queryNodes`,
-  which has no filter)
+  neighbours belong to another tenant (the behaviour of Cypher `db.index.vector.queryNodes`
+  without its `{where, filter}` options map, new in 26.11.1)
 - checks that every result row, and every fact reached by a hop, belongs to the tenant
 - builds a small synthetic memory (fixed-seed numpy embeddings); no data download
 - flags: `--form` (`where`, default, needs 26.11.1; or `filter`) and `--db-path`

src/arcadedb/_native/schema_objects.py

@@ -374,19 +374,25 @@ def _check_vector_arguments(
     rerank_oversample=None,
     fused_pq=None,
 ):
+    quantization_kind = quantization.upper() if quantization else ""
     if any(
         v is not None
         for v in (
             pq_subspaces,
             pq_clusters,
             pq_center_globally,
             pq_training_limit,
-            rerank_oversample,
             fused_pq,
         )
     ):
-        if not quantization or quantization.upper() != "PRODUCT":
+        if quantization_kind != "PRODUCT":
             raise ValueError("PQ parameters require quantization='PRODUCT'")
+    # PRODUCT and, from 26.11.1 (ArcadeData/arcadedb#10968), BINARY rerank on the stored
+    # vectors; every other quantization ignores the setting.
+    if rerank_oversample is not None and quantization_kind not in ("PRODUCT", "BINARY"):
+        raise ValueError(
+            "rerank_oversample requires quantization='PRODUCT' or 'BINARY'"
+        )
     if (
         encoding
         and encoding.upper() == "INT8"

src/arcadedb/core.py

@@ -1579,8 +1579,10 @@ def create_vector_index(
                 walking on the PQ codes scores again on the stored vectors
                 (maps to the index setting "rerankOversample"; the engine default
                 is 10, 0 returns the PQ scores alone, negative is refused).
-                Requires quantization="PRODUCT". A query can override it with the
-                `rerank` option of `vectorNeighbors`.
+                Requires quantization="PRODUCT", or "BINARY" (ArcadeDB 26.11.1 and
+                later: the search scores the one-bit codes and reranks
+                k * rerank_oversample candidates; at least one per neighbour). A
+                query can override it with the `rerank` option of `vectorNeighbors`.
             fused_pq: Whether the graph file stores the PQ codes of every node's
                 neighbours inline (jvector FusedPQ; maps to the index setting
                 "fusedPQ"; off by default). Requires quantization="PRODUCT".
@@ -1623,19 +1625,28 @@ def create_vector_index(
         try:
             import jpype
 
+            quantization_kind = quantization.upper() if quantization else ""
             if any(
                 val is not None
                 for val in (
                     pq_subspaces,
                     pq_clusters,
                     pq_center_globally,
                     pq_training_limit,
-                    rerank_oversample,
                     fused_pq,
                 )
             ):
-                if not quantization or quantization.upper() != "PRODUCT":
+                if quantization_kind != "PRODUCT":
                     raise ValueError("PQ parameters require quantization='PRODUCT'")
+            # The engine reranks on the stored vectors for PRODUCT and, from 26.11.1
+            # (ArcadeData/arcadedb#10968), BINARY; every other quantization ignores it.
+            if rerank_oversample is not None and quantization_kind not in (
+                "PRODUCT",
+                "BINARY",
+            ):
+                raise ValueError(
+                    "rerank_oversample requires quantization='PRODUCT' or 'BINARY'"
+                )
 
             java_schema = self.schema._java_schema
 

src/arcadedb/vector.py

@@ -624,8 +624,10 @@ def find_nearest(
         ``vectorNeighbors``: the walk scores on the in-memory PQ codes and the best
         ``k * rerankOversample`` candidates are re-ranked on the stored vectors, so
         the distances returned are exact and the ranking can differ from an exact
-        walk's. Every other quantization takes the exact walk. For the per-query
-        ``rerank`` option, use ``vectorNeighbors`` in SQL.
+        walk's. A ``BINARY`` index (ArcadeDB 26.11.1 and later) scores on its one-bit
+        codes and re-ranks ``k * rerankOversample`` candidates the same way. The
+        other quantizations take the exact walk. For the per-query ``rerank``
+        option, use ``vectorNeighbors`` in SQL.
 
         Args:
             query_vector: Query vector as Python list, NumPy array, or array-like

tests/conftest.py

@@ -124,8 +124,6 @@
     "test_vector_params_verification.py::TestVectorParams::test_jvm_heap_check": "reads java.lang.Runtime through JPype",
     "test_vector_params_verification.py::TestVectorParams::test_wrapper_default_matches_engine_default": "reads the engine default through java.lang.String and GlobalConfiguration",
     "test_vector_params_verification.py::TestVectorParams::test_omitted_max_connections_reaches_the_index": "reads the engine default through java.lang.String and GlobalConfiguration",
-    "test_vector_sql.py::TestVectorSearchOptionsSql::test_where_map_and_condition": "picks the expected behaviour by com.arcadedb.Constants.getRawVersion() through JPype",
-    "test_vector_sql.py::TestVectorSearchOptionsSql::test_where_combines_with_filter": "picks the expected behaviour by com.arcadedb.Constants.getRawVersion() through JPype",
     # Java objects built or read by the test
     "test_conversion_fixes.py::test_java_instants_convert_without_losing_microseconds": JAVA_CLASS,
     "test_type_conversion.py::test_offset_datetime_conversion": JAVA_CLASS,

tests/test_vector_product_quantization.py

@@ -335,8 +335,7 @@ def test_get_metadata_reports_the_engine_defaults(self, test_db, has_pq_rerank):
         assert meta["rerank_oversample"] == 10
         assert meta["fused_pq"] is False
 
-    @pytest.mark.parametrize("setting", ["rerank_oversample", "fused_pq"])
-    def test_new_settings_require_product_quantization(self, test_db, setting):
+    def test_fused_pq_requires_product_quantization(self, test_db):
         test_db.command("sql", "CREATE VERTEX TYPE PqVertex")
         test_db.command("sql", "CREATE PROPERTY PqVertex.embedding ARRAY_OF_FLOATS")
 
@@ -346,5 +345,128 @@ def test_new_settings_require_product_quantization(self, test_db, setting):
                 "embedding",
                 dimensions=DIMS,
                 quantization="INT8",
-                **{setting: 5 if setting == "rerank_oversample" else True},
+                fused_pq=True,
             )
+
+    @pytest.mark.parametrize("quantization", ["INT8", "NONE", None])
+    def test_rerank_oversample_refuses_unquantized_and_int8(
+        self, test_db, quantization
+    ):
+        """Only PRODUCT and BINARY rerank on the stored vectors (ArcadeData/arcadedb#10968)."""
+        test_db.command("sql", "CREATE VERTEX TYPE PqVertex")
+        test_db.command("sql", "CREATE PROPERTY PqVertex.embedding ARRAY_OF_FLOATS")
+
+        with pytest.raises(
+            arcadedb.ArcadeDBError, match="requires quantization='PRODUCT' or 'BINARY'"
+        ):
+            test_db.create_vector_index(
+                "PqVertex",
+                "embedding",
+                dimensions=DIMS,
+                quantization=quantization,
+                rerank_oversample=5,
+            )
+
+
+class TestBinaryRerankOversample:
+    """`rerank_oversample` on a BINARY index (ArcadeData/arcadedb#10968, ArcadeDB 26.11.1).
+
+    A BINARY index scores nodes on its one-bit codes and reranks `k * rerankOversample`
+    candidates on the stored vectors, the setting a PRODUCT index uses. The Python check
+    that only let PRODUCT carry it is gone; the engine decides, and `has_pq_rerank` is
+    false on an engine from before #10545, which refuses the key outright. What #10968
+    itself changed (how many candidates are reranked) is a recall difference of a few
+    percent that no deterministic assertion can pin; the engine's own tests cover it.
+    """
+
+    @staticmethod
+    def _make(db, **kwargs):
+        db.command("sql", "CREATE VERTEX TYPE BinVertex")
+        db.command("sql", "CREATE PROPERTY BinVertex.embedding ARRAY_OF_FLOATS")
+        return db.create_vector_index(
+            "BinVertex",
+            "embedding",
+            dimensions=DIMS,
+            distance_function="euclidean",
+            quantization="BINARY",
+            **kwargs,
+        )
+
+    def test_binary_accepts_rerank_oversample(self, test_db, has_pq_rerank):
+        if not has_pq_rerank:
+            with pytest.raises(
+                arcadedb.ArcadeDBError, match="Unsupported metadata key"
+            ):
+                self._make(test_db, rerank_oversample=25)
+            return
+
+        meta = self._make(test_db, rerank_oversample=25).get_metadata()
+
+        assert meta["quantization"] == "BINARY"
+        assert meta["rerank_oversample"] == 25
+
+    def test_binary_reports_the_default_rerank_oversample(self, test_db, has_pq_rerank):
+        meta = self._make(test_db).get_metadata()
+
+        if not has_pq_rerank:
+            assert "rerank_oversample" not in meta
+            return
+        assert meta["rerank_oversample"] == 10
+
+    def test_rerank_oversample_is_a_metadata_key_of_a_binary_statement(
+        self, test_db, has_pq_rerank
+    ):
+        test_db.command("sql", "CREATE VERTEX TYPE BinVertex")
+        test_db.command("sql", "CREATE PROPERTY BinVertex.embedding ARRAY_OF_FLOATS")
+        create = lambda: test_db.command(  # noqa: E731
+            "sql",
+            "CREATE INDEX ON BinVertex (embedding) LSM_VECTOR METADATA "
+            f'{{"dimensions": {DIMS}, "quantization": "BINARY", "rerankOversample": 7}}',
+        )
+        if not has_pq_rerank:
+            with pytest.raises(Exception, match="Unsupported metadata key"):
+                create()
+            return
+        create()
+
+        meta = test_db.schema.get_vector_index("BinVertex", "embedding").get_metadata()
+
+        assert meta["quantization"] == "BINARY"
+        assert meta["rerank_oversample"] == 7
+
+    @pytest.mark.parametrize("setting", ["pq_subspaces", "fused_pq"])
+    def test_pq_only_settings_still_need_product(self, test_db, setting):
+        with pytest.raises(arcadedb.ArcadeDBError, match="require quantization"):
+            self._make(test_db, **{setting: 8 if setting == "pq_subspaces" else True})
+
+    def test_binary_search_with_rerank_returns_exact_distances(
+        self, test_db, corpus, has_pq_rerank
+    ):
+        vectors, queries = corpus
+        index = self._make(
+            test_db, **({"rerank_oversample": 20} if has_pq_rerank else {})
+        )
+        with test_db.transaction():
+            for i, vec in enumerate(vectors):
+                test_db.command(
+                    "sql",
+                    "INSERT INTO BinVertex SET id = ?, embedding = ?",
+                    i,
+                    arcadedb.to_java_float_array(vec),
+                )
+        index.build_graph_now()
+        options = (
+            "{ efSearch: 100, rerank: 20 }" if has_pq_rerank else "{ efSearch: 100 }"
+        )
+
+        rows = test_db.query(
+            "sql",
+            "SELECT id, distance FROM "  # nosec B608 - test-owned option literals
+            f"(SELECT expand(vectorNeighbors('BinVertex[embedding]', ?, {K}, {options})))",  # nosec B608
+            arcadedb.to_java_float_array(queries[0]),
+        ).to_list()
+
+        _, exact = _exact_neighbors(vectors, queries[0])
+        assert len(rows) == K
+        for row in rows:
+            assert row["distance"] == pytest.approx(float(exact[row["id"]]), rel=1e-4)