diff --git a/docs/modules/ROOT/pages/how-to/vectors.adoc b/docs/modules/ROOT/pages/how-to/vectors.adoc index f492554..aa76dba 100644 --- a/docs/modules/ROOT/pages/how-to/vectors.adoc +++ b/docs/modules/ROOT/pages/how-to/vectors.adoc @@ -137,7 +137,7 @@ Bind the query vector as a parameter, as the block does, instead of writing the === openCypher -The Cypher procedure `db.index.vector.queryNodes(index, k, vector)` yields `node` and `score`. Its arguments are in that order, `k` before the vector. A higher `score` is closer. The procedure has no filter, so a `WHERE` after `YIELD` runs on the `k` rows only. +The Cypher procedure `db.index.vector.queryNodes(index, k, vector)` yields `node` and `score`. Its arguments are in that order, `k` before the vector. A higher `score` is closer. On ArcadeDB 26.11.1 and later a fourth argument, an options map, restricts the search before the top `k` is taken: `where` (a map of property to value, or a condition string) and `filter` (a list of nodes or record ids), for example `CALL db.index.vector.queryNodes('Doc[embedding]', 2, $vec, {where: {name: 'python'}})`. Without the map, and on an engine before 26.11.1, there is no filter, so a `WHERE` after `YIELD` runs on the `k` rows only. [source,python] ---- @@ -195,6 +195,8 @@ vector In SQL, the `filter` option of `vectorNeighbors` does the same with a list of record ids written in the statement. The `where` option tests a condition on each candidate as the search reaches it (a map of property to value, or a condition string). +On ArcadeDB 26.11.1 and later, a selective `where` on an indexed property uses the property index, in `vectorNeighbors` and in `db.index.vector.queryNodes` alike: the search starts walking the graph, and when the condition keeps few records it lists them through the index and scores them directly. A broad condition keeps walking. Index the property you filter on, for example `CREATE INDEX ON Doc (name) NOTUNIQUE`. A condition on a property without an index still loads candidates to test them, as before. The plan and its measurements are in https://github.com/ArcadeData/arcadedb/issues/10921[ArcadeData/arcadedb#10921]. + Filtering the rows afterwards is the simplest way. The search runs first and the filter second, so ask for more neighbours than you need. A filter that drops rows can leave fewer than you asked for. [source,python] @@ -247,7 +249,7 @@ INT8 True Two rules hold for the quantization arguments of the method: -* The `pq_*`, `rerank_oversample` and `fused_pq` arguments need `quantization="PRODUCT"`. The call raises `ValueError` otherwise. The same keys are `METADATA` keys in a statement (`pqSubspaces`, `pqClusters`, `pqCenterGlobally`, `pqTrainingLimit`, `rerankOversample`, `fusedPQ`). With `PRODUCT`, every graph rebuild retrains the codebook and re-encodes every vector (https://github.com/ArcadeData/arcadedb/issues/10847[ArcadeData/arcadedb#10847]), so measure recall and rebuild time on your own data before you choose it. +* The `pq_*` and `fused_pq` arguments need `quantization="PRODUCT"`. `rerank_oversample` needs `"PRODUCT"` or, on ArcadeDB 26.11.1 and later, `"BINARY"`: a `BINARY` search scores the one-bit codes and re-ranks `k * rerank_oversample` candidates on the stored vectors (default 10), or the whole `efSearch` beam when that is larger (https://github.com/ArcadeData/arcadedb/issues/10968[ArcadeData/arcadedb#10968]). The call raises `ValueError` otherwise. The same keys are `METADATA` keys in a statement (`pqSubspaces`, `pqClusters`, `pqCenterGlobally`, `pqTrainingLimit`, `rerankOversample`, `fusedPQ`). With `PRODUCT`, every graph rebuild retrains the codebook and re-encodes every vector (https://github.com/ArcadeData/arcadedb/issues/10847[ArcadeData/arcadedb#10847]), so measure recall and rebuild time on your own data before you choose it. * `encoding="INT8"` is for a `BINARY` property that already holds INT8 bytes. Set `quantization` to `"NONE"` with it. The call raises `ValueError` when both are `INT8`, because the vector would be quantized twice. == Rebuild the graph diff --git a/examples/29_hybrid_memory_retrieval.py b/examples/29_hybrid_memory_retrieval.py index c184b4b..eecc055 100644 --- a/examples/29_hybrid_memory_retrieval.py +++ b/examples/29_hybrid_memory_retrieval.py @@ -15,8 +15,9 @@ The pre-filter is the point. Taking the global top-k first and filtering by tenant afterwards returns fewer rows than asked, or none, whenever the nearest neighbours belong to another tenant (this is what Cypher's -``db.index.vector.queryNodes(index, k, vector)`` does: it has no filter, so -``WHERE node.tenant = ...`` runs on its k results; ArcadeData/arcadedb#10685). +``db.index.vector.queryNodes(index, k, vector)`` does without its fourth +argument, the ``{where, filter}`` options map of ArcadeDB 26.11.1: a ``WHERE`` +after ``YIELD`` runs on its k results; ArcadeData/arcadedb#10685). ``vector.neighbors`` takes the restriction as an option, so the index search itself only considers the tenant's vectors: @@ -29,7 +30,9 @@ has ``where``. The index types are created with SQL and are always the same: an ``LSM_VECTOR`` -index on ``Fact.embedding`` and a ``FULL_TEXT`` index on ``Fact.text``. +index on ``Fact.embedding`` and a ``FULL_TEXT`` index on ``Fact.text``, plus +a ``NOTUNIQUE`` index on ``Fact.tenant``: from 26.11.1 a selective ``where`` on +an indexed property takes the property index instead of loading candidates. ``vector.fuse`` and the ``SEARCH_INDEX`` source are used as documented in the ArcadeDB vector-search concept page ("Hybrid Search with vector.fuse"). @@ -172,6 +175,7 @@ def build(db) -> dict: "CREATE INDEX ON Fact (fid) UNIQUE_HASH", "CREATE INDEX ON Entity (eid) UNIQUE_HASH", "CREATE INDEX ON Fact (text) FULL_TEXT", + "CREATE INDEX ON Fact (tenant) NOTUNIQUE", 'CREATE INDEX ON Fact (embedding) LSM_VECTOR METADATA {"dimensions": %d, ' '"similarity": "COSINE"}' % DIM, ): diff --git a/examples/README.md b/examples/README.md index aa9273a..eb99c85 100644 --- a/examples/README.md +++ b/examples/README.md @@ -430,8 +430,8 @@ Hybrid retrieval for an agent memory in one SQL statement instead of three store the same tenant, both fused by reciprocal rank fusion with `vector.fuse`, then graph hops (`MENTIONS`, `RELATES`) to related entities and facts, kept inside the tenant - shows why: a global top-k followed by a tenant test returns 0 rows when the nearest - neighbours belong to another tenant (the behaviour of Cypher `db.index.vector.queryNodes`, - which has no filter) + neighbours belong to another tenant (the behaviour of Cypher `db.index.vector.queryNodes` + without its `{where, filter}` options map, new in 26.11.1) - checks that every result row, and every fact reached by a hop, belongs to the tenant - builds a small synthetic memory (fixed-seed numpy embeddings); no data download - flags: `--form` (`where`, default, needs 26.11.1; or `filter`) and `--db-path` diff --git a/src/arcadedb/_native/schema_objects.py b/src/arcadedb/_native/schema_objects.py index eb53a7d..52a85e7 100644 --- a/src/arcadedb/_native/schema_objects.py +++ b/src/arcadedb/_native/schema_objects.py @@ -374,6 +374,7 @@ def _check_vector_arguments( rerank_oversample=None, fused_pq=None, ): + quantization_kind = quantization.upper() if quantization else "" if any( v is not None for v in ( @@ -381,12 +382,17 @@ def _check_vector_arguments( pq_clusters, pq_center_globally, pq_training_limit, - rerank_oversample, fused_pq, ) ): - if not quantization or quantization.upper() != "PRODUCT": + if quantization_kind != "PRODUCT": raise ValueError("PQ parameters require quantization='PRODUCT'") + # PRODUCT and, from 26.11.1 (ArcadeData/arcadedb#10968), BINARY rerank on the stored + # vectors; every other quantization ignores the setting. + if rerank_oversample is not None and quantization_kind not in ("PRODUCT", "BINARY"): + raise ValueError( + "rerank_oversample requires quantization='PRODUCT' or 'BINARY'" + ) if ( encoding and encoding.upper() == "INT8" diff --git a/src/arcadedb/core.py b/src/arcadedb/core.py index 4309ac0..47955e8 100644 --- a/src/arcadedb/core.py +++ b/src/arcadedb/core.py @@ -1579,8 +1579,10 @@ def create_vector_index( walking on the PQ codes scores again on the stored vectors (maps to the index setting "rerankOversample"; the engine default is 10, 0 returns the PQ scores alone, negative is refused). - Requires quantization="PRODUCT". A query can override it with the - `rerank` option of `vectorNeighbors`. + Requires quantization="PRODUCT", or "BINARY" (ArcadeDB 26.11.1 and + later: the search scores the one-bit codes and reranks + k * rerank_oversample candidates; at least one per neighbour). A + query can override it with the `rerank` option of `vectorNeighbors`. fused_pq: Whether the graph file stores the PQ codes of every node's neighbours inline (jvector FusedPQ; maps to the index setting "fusedPQ"; off by default). Requires quantization="PRODUCT". @@ -1623,6 +1625,7 @@ def create_vector_index( try: import jpype + quantization_kind = quantization.upper() if quantization else "" if any( val is not None for val in ( @@ -1630,12 +1633,20 @@ def create_vector_index( pq_clusters, pq_center_globally, pq_training_limit, - rerank_oversample, fused_pq, ) ): - if not quantization or quantization.upper() != "PRODUCT": + if quantization_kind != "PRODUCT": raise ValueError("PQ parameters require quantization='PRODUCT'") + # The engine reranks on the stored vectors for PRODUCT and, from 26.11.1 + # (ArcadeData/arcadedb#10968), BINARY; every other quantization ignores it. + if rerank_oversample is not None and quantization_kind not in ( + "PRODUCT", + "BINARY", + ): + raise ValueError( + "rerank_oversample requires quantization='PRODUCT' or 'BINARY'" + ) java_schema = self.schema._java_schema diff --git a/src/arcadedb/vector.py b/src/arcadedb/vector.py index fea4e62..5ac925f 100644 --- a/src/arcadedb/vector.py +++ b/src/arcadedb/vector.py @@ -624,8 +624,10 @@ def find_nearest( ``vectorNeighbors``: the walk scores on the in-memory PQ codes and the best ``k * rerankOversample`` candidates are re-ranked on the stored vectors, so the distances returned are exact and the ranking can differ from an exact - walk's. Every other quantization takes the exact walk. For the per-query - ``rerank`` option, use ``vectorNeighbors`` in SQL. + walk's. A ``BINARY`` index (ArcadeDB 26.11.1 and later) scores on its one-bit + codes and re-ranks ``k * rerankOversample`` candidates the same way. The + other quantizations take the exact walk. For the per-query ``rerank`` + option, use ``vectorNeighbors`` in SQL. Args: query_vector: Query vector as Python list, NumPy array, or array-like diff --git a/tests/conftest.py b/tests/conftest.py index c49cf70..1eec1b7 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -124,8 +124,6 @@ "test_vector_params_verification.py::TestVectorParams::test_jvm_heap_check": "reads java.lang.Runtime through JPype", "test_vector_params_verification.py::TestVectorParams::test_wrapper_default_matches_engine_default": "reads the engine default through java.lang.String and GlobalConfiguration", "test_vector_params_verification.py::TestVectorParams::test_omitted_max_connections_reaches_the_index": "reads the engine default through java.lang.String and GlobalConfiguration", - "test_vector_sql.py::TestVectorSearchOptionsSql::test_where_map_and_condition": "picks the expected behaviour by com.arcadedb.Constants.getRawVersion() through JPype", - "test_vector_sql.py::TestVectorSearchOptionsSql::test_where_combines_with_filter": "picks the expected behaviour by com.arcadedb.Constants.getRawVersion() through JPype", # Java objects built or read by the test "test_conversion_fixes.py::test_java_instants_convert_without_losing_microseconds": JAVA_CLASS, "test_type_conversion.py::test_offset_datetime_conversion": JAVA_CLASS, diff --git a/tests/test_vector_product_quantization.py b/tests/test_vector_product_quantization.py index 5d4ac31..3b85657 100644 --- a/tests/test_vector_product_quantization.py +++ b/tests/test_vector_product_quantization.py @@ -335,8 +335,7 @@ def test_get_metadata_reports_the_engine_defaults(self, test_db, has_pq_rerank): assert meta["rerank_oversample"] == 10 assert meta["fused_pq"] is False - @pytest.mark.parametrize("setting", ["rerank_oversample", "fused_pq"]) - def test_new_settings_require_product_quantization(self, test_db, setting): + def test_fused_pq_requires_product_quantization(self, test_db): test_db.command("sql", "CREATE VERTEX TYPE PqVertex") test_db.command("sql", "CREATE PROPERTY PqVertex.embedding ARRAY_OF_FLOATS") @@ -346,5 +345,128 @@ def test_new_settings_require_product_quantization(self, test_db, setting): "embedding", dimensions=DIMS, quantization="INT8", - **{setting: 5 if setting == "rerank_oversample" else True}, + fused_pq=True, ) + + @pytest.mark.parametrize("quantization", ["INT8", "NONE", None]) + def test_rerank_oversample_refuses_unquantized_and_int8( + self, test_db, quantization + ): + """Only PRODUCT and BINARY rerank on the stored vectors (ArcadeData/arcadedb#10968).""" + test_db.command("sql", "CREATE VERTEX TYPE PqVertex") + test_db.command("sql", "CREATE PROPERTY PqVertex.embedding ARRAY_OF_FLOATS") + + with pytest.raises( + arcadedb.ArcadeDBError, match="requires quantization='PRODUCT' or 'BINARY'" + ): + test_db.create_vector_index( + "PqVertex", + "embedding", + dimensions=DIMS, + quantization=quantization, + rerank_oversample=5, + ) + + +class TestBinaryRerankOversample: + """`rerank_oversample` on a BINARY index (ArcadeData/arcadedb#10968, ArcadeDB 26.11.1). + + A BINARY index scores nodes on its one-bit codes and reranks `k * rerankOversample` + candidates on the stored vectors, the setting a PRODUCT index uses. The Python check + that only let PRODUCT carry it is gone; the engine decides, and `has_pq_rerank` is + false on an engine from before #10545, which refuses the key outright. What #10968 + itself changed (how many candidates are reranked) is a recall difference of a few + percent that no deterministic assertion can pin; the engine's own tests cover it. + """ + + @staticmethod + def _make(db, **kwargs): + db.command("sql", "CREATE VERTEX TYPE BinVertex") + db.command("sql", "CREATE PROPERTY BinVertex.embedding ARRAY_OF_FLOATS") + return db.create_vector_index( + "BinVertex", + "embedding", + dimensions=DIMS, + distance_function="euclidean", + quantization="BINARY", + **kwargs, + ) + + def test_binary_accepts_rerank_oversample(self, test_db, has_pq_rerank): + if not has_pq_rerank: + with pytest.raises( + arcadedb.ArcadeDBError, match="Unsupported metadata key" + ): + self._make(test_db, rerank_oversample=25) + return + + meta = self._make(test_db, rerank_oversample=25).get_metadata() + + assert meta["quantization"] == "BINARY" + assert meta["rerank_oversample"] == 25 + + def test_binary_reports_the_default_rerank_oversample(self, test_db, has_pq_rerank): + meta = self._make(test_db).get_metadata() + + if not has_pq_rerank: + assert "rerank_oversample" not in meta + return + assert meta["rerank_oversample"] == 10 + + def test_rerank_oversample_is_a_metadata_key_of_a_binary_statement( + self, test_db, has_pq_rerank + ): + test_db.command("sql", "CREATE VERTEX TYPE BinVertex") + test_db.command("sql", "CREATE PROPERTY BinVertex.embedding ARRAY_OF_FLOATS") + create = lambda: test_db.command( # noqa: E731 + "sql", + "CREATE INDEX ON BinVertex (embedding) LSM_VECTOR METADATA " + f'{{"dimensions": {DIMS}, "quantization": "BINARY", "rerankOversample": 7}}', + ) + if not has_pq_rerank: + with pytest.raises(Exception, match="Unsupported metadata key"): + create() + return + create() + + meta = test_db.schema.get_vector_index("BinVertex", "embedding").get_metadata() + + assert meta["quantization"] == "BINARY" + assert meta["rerank_oversample"] == 7 + + @pytest.mark.parametrize("setting", ["pq_subspaces", "fused_pq"]) + def test_pq_only_settings_still_need_product(self, test_db, setting): + with pytest.raises(arcadedb.ArcadeDBError, match="require quantization"): + self._make(test_db, **{setting: 8 if setting == "pq_subspaces" else True}) + + def test_binary_search_with_rerank_returns_exact_distances( + self, test_db, corpus, has_pq_rerank + ): + vectors, queries = corpus + index = self._make( + test_db, **({"rerank_oversample": 20} if has_pq_rerank else {}) + ) + with test_db.transaction(): + for i, vec in enumerate(vectors): + test_db.command( + "sql", + "INSERT INTO BinVertex SET id = ?, embedding = ?", + i, + arcadedb.to_java_float_array(vec), + ) + index.build_graph_now() + options = ( + "{ efSearch: 100, rerank: 20 }" if has_pq_rerank else "{ efSearch: 100 }" + ) + + rows = test_db.query( + "sql", + "SELECT id, distance FROM " # nosec B608 - test-owned option literals + f"(SELECT expand(vectorNeighbors('BinVertex[embedding]', ?, {K}, {options})))", # nosec B608 + arcadedb.to_java_float_array(queries[0]), + ).to_list() + + _, exact = _exact_neighbors(vectors, queries[0]) + assert len(rows) == K + for row in rows: + assert row["distance"] == pytest.approx(float(exact[row["id"]]), rel=1e-4) diff --git a/tests/test_vector_sql.py b/tests/test_vector_sql.py index 84771d9..c752cdb 100644 --- a/tests/test_vector_sql.py +++ b/tests/test_vector_sql.py @@ -1120,16 +1120,18 @@ def test_find_nearest_repeated_calls_consistent(self, test_db): assert ids1 == ids2 and len(ids1) == 3 -def _engine_version(): - import jpype - - raw = str(jpype.JClass("com.arcadedb.Constants").getRawVersion()) - parts = [int(p) if p.isdigit() else 0 for p in raw.split("-")[0].split(".")[:3]] +def _engine_version(db): + """The engine's x.y.z through SQL `SELECT version()`, so the same check runs on the JVM and the native build.""" + raw = str(db.query("sql", "SELECT version() AS v").first().get("v")) + parts = [ + int(p) if p.isdigit() else 0 + for p in raw.split()[0].split("-")[0].split(".")[:3] + ] return tuple(parts + [0] * (3 - len(parts))) -# `where` in the vector.neighbors options is new in ArcadeDB 26.11.1. The JVM only runs -# once a database fixture has started it, so the version is read inside each test, and +# `where` in the vector.neighbors options is new in ArcadeDB 26.11.1. The version is read +# through the test's database inside each test (both builds), and # both engines are asserted on (the repository accepts no skips, scripts/check_test_skips.py). WHERE_SINCE = (26, 11, 1) @@ -1207,7 +1209,7 @@ def test_group_by_with_group_size_two(self, test_db): def test_where_map_and_condition(self, test_db): self._make_docs(test_db) - if _engine_version() < WHERE_SINCE: + if _engine_version(test_db) < WHERE_SINCE: with pytest.raises(Exception, match="Unknown option 'where'"): self._search(test_db, "{ where: { lang: 'fr' } }") return @@ -1226,7 +1228,7 @@ def test_where_combines_with_filter(self, test_db): allowed = f"['{rids[2]}', '{rids[3]}', '{rids[5]}']" options = f"{{ filter: {allowed}, where: {{ lang: 'fr' }} }}" - if _engine_version() < WHERE_SINCE: + if _engine_version(test_db) < WHERE_SINCE: with pytest.raises(Exception, match="Unknown option 'where'"): self._search(test_db, options) return @@ -1235,6 +1237,65 @@ def test_where_combines_with_filter(self, test_db): assert [r["id"] for r in rows] == [3, 5] + @staticmethod + def _make_tenants(db): + """200 vertices; `tenant` is indexed (NOTUNIQUE), `lang` is not.""" + db.command("sql", "CREATE VERTEX TYPE QnDoc") + db.command("sql", "CREATE PROPERTY QnDoc.id INTEGER") + db.command("sql", "CREATE PROPERTY QnDoc.tenant INTEGER") + db.command("sql", "CREATE PROPERTY QnDoc.lang STRING") + db.command("sql", "CREATE PROPERTY QnDoc.embedding ARRAY_OF_FLOATS") + db.command("sql", "CREATE INDEX ON QnDoc (tenant) NOTUNIQUE") + db.command( + "sql", + "CREATE INDEX ON QnDoc (embedding) LSM_VECTOR METADATA " + '{"dimensions": 2, "similarity": "EUCLIDEAN"}', + ) + with db.transaction(): + for i in range(200): + db.command( + "sql", + "INSERT INTO QnDoc SET id = ?, tenant = ?, lang = ?, embedding = ?", + i, + 1 if i % 50 == 7 else 0, + "en" if i % 2 == 0 else "fr", + arcadedb.to_java_float_array([float(i), 0.0]), + ) + + @staticmethod + def _query_nodes(db, options, k=10): + return [ + r["id"] + for r in db.query( + "opencypher", + f"CALL db.index.vector.queryNodes('QnDoc[embedding]', {k}, $vec, {options}) " # nosec B608 + "YIELD node, score RETURN node.id AS id ORDER BY score DESC", + {"vec": [0.0, 0.0]}, + ).to_list() + ] + + def test_query_nodes_takes_the_where_options_map(self, test_db): + """`db.index.vector.queryNodes(index, k, vector, {where: ...})`: ArcadeDB 26.11.1. + + A selective `where` on the indexed `tenant` takes the property index + (ArcadeData/arcadedb#10957); the answer is the same rows whichever plan + runs, so this pins the rows, not the plan. `lang` has no index. + """ + self._make_tenants(test_db) + + if _engine_version(test_db) < WHERE_SINCE: + with pytest.raises(Exception, match="expects 3 arguments but got 4"): + self._query_nodes(test_db, "{where: {tenant: 1}}") + return + + by_map = self._query_nodes(test_db, "{where: {tenant: 1}}") + by_condition = self._query_nodes(test_db, "{where: 'tenant = 1'}") + unindexed = self._query_nodes(test_db, "{where: {lang: 'fr'}}") + + assert sorted(by_map) == [7, 57, 107, 157] + assert sorted(by_condition) == [7, 57, 107, 157] + assert unindexed == [1, 3, 5, 7, 9, 11, 13, 15, 17, 19] + def test_unknown_option_lists_the_allowed_ones(self, test_db): self._make_docs(test_db)