diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dc6b96bed..98e6f2ac1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -229,7 +229,7 @@ The Plugin API is the primary way to implement additional functionality in Harpe - `logger.level` — Log level; _Default_: `"info"` (Added in: v4.1.0) ``` -For inline config option annotations inside list items, plain text `(Added in: vX.Y.Z)` is fine — using the component mid-sentence is awkward. Reserve `` for standalone placement after headings. +Use `` after headings or alongside the specific sentence, list item, or table entry it annotates. Use `` the same way when behavior is storage-engine-specific; comma-separated values such as `engines="RocksDB, LMDB"` are supported. Keep adjacent version and engine badges together on one line. For dense prose where a component would interrupt the sentence, plain text such as `(Added in: vX.Y.Z)` is fine. ## Redirects diff --git a/learn/developers/full-text-search-product-catalog.mdx b/learn/developers/full-text-search-product-catalog.mdx new file mode 100644 index 000000000..3be7b7ee2 --- /dev/null +++ b/learn/developers/full-text-search-product-catalog.mdx @@ -0,0 +1,193 @@ +--- +title: Add Full-Text Search to a Product Catalog +description: Configure and query a BM25-ranked product search in Harper. +--- + +# Add Full-Text Search to a Product Catalog + + + +This guide adds relevance-ranked text search to a product catalog, then exposes it through Harper's JavaScript and REST APIs. + +## What you will build + +- A BM25-ranked index over product names, descriptions, and tags +- A Harper application endpoint with highlights and a structured category filter +- REST queries for terms, phrases, and record autocomplete + +## Prerequisites + +- Harper 5.3 or later using RocksDB +- A component with a GraphQL schema, JavaScript resources, and `rest: true` in its `config.yaml` + +## 1. Declare the index + +Declare an `@fullText` index over the fields customers search on a RocksDB-backed table. Ordinary tables inherit enabled audit logging by default, but a new full-text table requires explicit `audit: true` for its transaction-log recovery source. See the [audit requirement](/reference/v5/full-text-search/configuration) for existing tables. + +```graphql title="schema.graphql" +type Product @table(database: "catalog", audit: true) @export { + id: ID @primaryKey + name: String + description: String + tags: [String] + category: String @indexed + price: Float @indexed + available: Boolean + catalogSearch: FullText + @fullText( + fields: [ + { name: "name", weight: 3, highlight: true } + { name: "description", weight: 1, highlight: true } + { name: "tags" } + ] + highlighting: { maxFragments: 2, fragmentLength: 120 } + ) +} +``` + +The query-only field name `catalogSearch` identifies the index. It is not stored with each product. `weight: 3` makes a match in `name` contribute more to BM25 relevance than the same match in `description` or `tags`. Highlighting is enabled only for the fields shown to customers. + +The `@indexed` declarations create ordinary Harper secondary indexes. Full-text searches can use them to narrow candidates, and Harper checks structured conditions against current product records before returning results. + +## 2. Add records + +Use the generated REST endpoint: + +```http +POST /Product/ +Content-Type: application/json + +{ + "id": "shoe-1", + "name": "Waterproof trail running shoe", + "description": "Lightweight shoe for wet mountain trails", + "tags": ["outdoor", "trail"], + "category": "footwear", + "price": 129, + "available": true +} +``` + +Harper commits the record first, then applies the committed change to the local derived full-text index. Every replica performs the same derivation from the transaction it receives. + +If the index remains behind for more than 30 seconds, a local write can return retryable `503` code `DERIVED_INDEX_LAGGING`. Retry writes with backoff while the index catches up; freshness options apply only to searches. See [write backpressure](/reference/v5/full-text-search/overview#write-backpressure). + +## 3. Search from a Harper application + +```javascript title="resources.js" +export class SearchProducts extends Resource { + static async post(_target, data, context) { + if (!context?.user) { + const error = new Error('Authentication required'); + error.statusCode = 401; + throw error; + } + const body = (await data) ?? {}; + const text = typeof body.q === 'string' ? body.q.trim() : ''; + if (!text) { + const error = new Error('q is required'); + error.statusCode = 400; + throw error; + } + const conditions = [ + { + attribute: 'catalogSearch', + comparator: 'matches', + value: text, + fields: ['name', 'description', 'tags'], + }, + { attribute: 'available', comparator: 'equals', value: true }, + ]; + const category = body.category; + if (category) conditions.push({ attribute: 'category', comparator: 'equals', value: category }); + return databases.catalog.Product.search( + { + conditions, + select: ['id', 'name', 'price', '$score', '$highlights'], + limit: 20, + checkPermission: true, + }, + context + ); + } +} +``` + +Call the resource with an authenticated request: + +```http +POST /SearchProducts/ +Content-Type: application/json + +{ "q": "trail", "category": "footwear" } +``` + +This custom resource accepts a structured JSON search body through POST. The category filter is optional; `q` is required. For URL-based GET searches, use the exported table endpoint in step 4. + +Harper applies the availability and optional category conditions while accepting current product records. Existing secondary indexes can narrow candidates; rejected records do not count toward the requested page. + +Calls through `tables` and `databases` use Harper's [trusted server-side context](/reference/v5/components/javascript-environment#tables). Forwarding the resource's `context` preserves the authenticated caller, and `checkPermission: true` asks the table to enforce that caller's table and source-field permissions. The custom resource must do both. Highlights can expose searched source text even when those fields are absent from `select`. + +Matches in `tags` contribute to search results and ranking, but `$highlights` does not include tag matches because `tags` is not configured with `highlight: true`. + +## 4. Search through REST + +The exported table supports the same match comparator: + +```http +GET /Product/?catalogSearch=matches=waterproof%20trail&category=footwear&select(id,name,price,$score,$highlights)&limit(20) +``` + +For a phrase: + +```http +GET /Product/?catalogSearch=matches_phrase=trail%20running&limit(20) +``` + +For record autocomplete: + +```http +GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&select(id,name)&limit(10) +``` + +Prefix search returns matching products. It does not return a dictionary of suggested terms. Before v5.4, prefix and fuzzy-prefix queries restrict `offset + limit` to 100. In v5.4, bounded pages can go beyond that window within request budgets; each page runs a new query rather than retaining a search. See [paging and prefix result windows](/reference/v5/full-text-search/querying#paging-and-prefix-result-windows). + +REST searches every source field in the index, so the caller must be allowed to read `name`, `description`, and `tags` for these requests. Use `Table.search()` or `search_by_conditions` when a role should search only a subset of sources. See [REST full-text operators](/reference/v5/rest/querying#full-text-operators). + +## 5. Choose consistency behavior + +Most searches can use the default three-second lag tolerance. When the next request must find a record that was just written, use the JavaScript API and wait for current index coverage: + +```javascript +for await (const product of databases.catalog.Product.search({ + conditions: [ + { + attribute: 'catalogSearch', + comparator: 'matches', + value: 'new seasonal product', + maxIndexLagMilliseconds: 0, + waitForIndexMilliseconds: 10000, + }, + ], +})) { + console.log(product); +} +``` + +Do not apply this wait to every catalog request without measuring it. The normal bounded-staleness path keeps search traffic independent of short indexing bursts. + +If the index does not reach current coverage within ten seconds, iteration throws an error with `statusCode: 503`, `code: "DERIVED_INDEX_LAGGING"`, and `retryable: true`. See [freshness controls](/reference/v5/full-text-search/querying#freshness-controls). + +## 6. Operate the index + +Use `describe_table` to inspect readiness. Harper reuses compatible local index files after restart and replays changes after their checkpoint. A node rebuilds locally when the files are missing, incompatible, or corrupt; table traffic remains available, while full-text requests in `unknown`, `needs-rebuild`, or `rebuilding` state return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. Retry this code with backoff; do not treat every `503` as an index rebuild. + +The custom resource in step 3 streams its returned iterable. HTTP error statuses apply before response headers are sent; a failure after streaming starts is a terminal stream error and cannot change the HTTP status. Consumers must also handle that terminal error. See [freshness and streaming errors](/reference/v5/full-text-search/querying#freshness-controls). + +See the [full-text configuration reference](/reference/v5/full-text-search/configuration) and [query reference](/reference/v5/full-text-search/querying) for every option and match mode. + +## Additional resources + +- [Full-text search overview](/reference/v5/full-text-search/overview) +- [Resource API](/reference/v5/resources/resource-api) +- [Query optimization](/reference/v5/resources/query-optimization) diff --git a/reference/backups/overview.md b/reference/backups/overview.md index 5596ebe28..7d812effc 100644 --- a/reference/backups/overview.md +++ b/reference/backups/overview.md @@ -5,7 +5,7 @@ title: Backups -Harper can back up and restore its databases natively. A backup is a whole-database copy — all tables, the audit/transaction log, and any file-backed blobs — so a restored database is a consistent point-in-time image of everything in it. (`get_backup` on an LMDB database is the exception: it can stream a subset of tables, and includes the audit store only when requested.) +Harper can back up and restore its databases natively. A backup is a whole-database copy of authoritative state — all tables, the audit/transaction log, and any file-backed blobs — so a restored database is a consistent point-in-time image. Local derived state, including full-text index files, is excluded and rebuilt after restore. (`get_backup` on an LMDB database is the exception: it can stream a subset of tables, and includes the audit store only when requested.) Two complementary mechanisms are available: @@ -38,6 +38,7 @@ The offline path matters for restore. RocksDB is single-writer, so an in-place r - **Backups live on the node that created them.** The backup repository is a local directory. RocksDB shares files across backup IDs (a backup ID is not a self-contained folder), so disaster-recovery copies must take the entire per-database repository — `/` — not an individual backup, and must do so while no backup operation is running, or from an atomic filesystem snapshot. A live recursive copy can race `create_backup`/`delete_backup`/`purge_backups` and produce an unrestorable copy. Alternatively, use `get_backup` to pull a snapshot from a running server. - **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its whole per-database repository as above. - **A restore is a point-in-time rollback.** In a replicated cluster, coordinate a restore with replication before bringing the node back. +- **Full-text indexes rebuild locally after restore.** Ordinary table traffic remains available. While the rebuild is active, full-text requests return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`; branch on the code because other `503` responses are not necessarily rebuilds. - **A restore checks the backup before it deletes anything.** A backup with a corrupt database file, or one this version of Harper cannot read, is refused and the database is left as it was (a database an earlier restore left incomplete stays blocked). The check reads the whole backup, which adds to how long an online restore keeps the database unavailable. Blob file contents are not checked. See [`restore_backup`](./operations.md#restore_backup). - **An interrupted restore leaves the database unloadable.** If `restore_backup` is interrupted after it has started replacing the database's files (crash, power loss), Harper marks the database as incompletely restored and skips loading it on the next start, logging an incomplete-restore error. Rerun `restore_backup` for the same database and `backup_id` to recover; do not load or hand-repair the directory. diff --git a/reference/components/javascript-environment.md b/reference/components/javascript-environment.md index 78f626b60..d26a575a6 100644 --- a/reference/components/javascript-environment.md +++ b/reference/components/javascript-environment.md @@ -90,7 +90,7 @@ An object whose properties are the tables in the default database (`data`). Each See [Database API](../database/api.md) for full reference. :::note Server-side table reads run in a trusted context -Calls to `tables.X.get()`, `tables.X.search()`, and equivalent `databases..` methods from within resource code execute in a **trusted system context** and do **not** re-apply the target table's `allowRead` guard or role-level `attribute_permissions`. This is by design — server-side code is responsible for its own authorization checks. A computed attribute or custom resource that cross-reads a protected table can return raw data regardless of the caller's role. Do not expose protected data through a computed attribute when access depends on the caller. Instead, use a custom resource and apply authorization explicitly (e.g. call `this.getCurrentUser()` in a `Resource` method, or use `getUser()`, and enforce the relevant restrictions before returning the data). +Calls to `tables.X.get()`, `tables.X.search()`, and equivalent `databases..
` methods from within resource code execute in a **trusted system context** and do **not** re-apply the target table's `allowRead` guard or role-level `attribute_permissions` by default. This is by design — server-side code is responsible for its own authorization checks. A custom resource can explicitly re-enter table and source-field authorization for `Table.search()` by forwarding its authenticated `context` and setting `checkPermission: true` in search options. Never accept or copy that flag or a permission object from client data. Context propagation alone does not request the check, and other direct table operations still require explicit authorization. A computed attribute or custom resource that cross-reads a protected table can otherwise return raw data regardless of the caller's role. Do not expose protected data through a computed attribute when access depends on the caller. Instead, use a custom resource and enforce the relevant restrictions before returning the data. ::: ### `databases` diff --git a/reference/database/overview.md b/reference/database/overview.md index 406b251a9..bfa0434d2 100644 --- a/reference/database/overview.md +++ b/reference/database/overview.md @@ -107,6 +107,7 @@ It is best practice to define schemas for production tables. Dynamic schemas are For deeper coverage of each database feature, see the dedicated pages in this section: - **[Schema](./schema.md)** — Defining table structure, types, indexes, relationships, and computed properties using GraphQL schema syntax +- **[Full-Text Search](../full-text-search/overview.md)**: BM25-ranked search over stored text fields on audited RocksDB tables - **[API](./api.md)** — The `tables`, `databases`, `transaction()`, and `createBlob()` globals for interacting with the database from code - **[Data Loader](./data-loader.md)** — Loading seed or initial data into tables as part of component deployment - **[Storage Algorithm](./storage-algorithm.md)** — How Harper stores data using LMDB with universal indexing and ACID compliance diff --git a/reference/database/schema.md b/reference/database/schema.md index 43d9de262..f2c0b3545 100644 --- a/reference/database/schema.md +++ b/reference/database/schema.md @@ -73,16 +73,17 @@ type MyTable @table { Optional arguments: -| Argument | Type | Default | Description | -| -------------------- | --------- | ----------------------------- | ------------------------------------------------------------------------------------------- | -| `table` | `String` | type name | Override the table name | -| `database` | `String` | `"data"` | Database to place the table in | -| `expiration` | `Int` | — | Seconds until a record goes stale (useful for caching tables) | -| `eviction` | `Int` | `0` | Additional seconds after `expiration` before a record is physically removed | -| `scanInterval` | `Int` | `(expiration + eviction) / 4` | Seconds between eviction scans | -| `replicate` | `Boolean` | true | Enable replication of this table | -| `cacheControl` | `String` | — | `Cache-Control` header value emitted on anonymous GET/HEAD 200/304 responses for this table | -| `randomAccessFields` | `Boolean` | `storage.randomAccessFields` | [Pin this table's record encoding](#randomaccessfields) | +| Argument | Type | Default | Description | +| -------------------- | --------- | -------------------------------------- | ------------------------------------------------------------------------------------------- | +| `table` | `String` | type name | Override the table name | +| `database` | `String` | `"data"` | Database to place the table in | +| `expiration` | `Int` | — | Seconds until a record goes stale (useful for caching tables) | +| `eviction` | `Int` | `0` | Additional seconds after `expiration` before a record is physically removed | +| `scanInterval` | `Int` | `(expiration + eviction) / 4` | Seconds between eviction scans | +| `replicate` | `Boolean` | true | Enable replication of this table | +| `audit` | `Boolean` | `logging.auditLog` for ordinary tables | Enable the table's transaction log; new tables with `@fullText` require explicit `true` | +| `cacheControl` | `String` | — | `Cache-Control` header value emitted on anonymous GET/HEAD 200/304 responses for this table | +| `randomAccessFields` | `Boolean` | `storage.randomAccessFields` | [Pin this table's record encoding](#randomaccessfields) | **`expiration`, `eviction`, and `scanInterval`** @@ -363,6 +364,24 @@ Write semantics: Multiple `@embed` attributes on one type are computed concurrently. +### `@fullText` + + + +Creates a BM25-ranked full-text derived index from one or more stored text fields. Declare the directive on a nullable `FullText` field; that field's name identifies the index. + +```graphql +type Product @table(database: "catalog", audit: true) { + id: ID @primaryKey + name: String + description: String + tags: [String] + catalogSearch: FullText @fullText(fields: [{ name: "name", weight: 3 }, { name: "description" }, { name: "tags" }]) +} +``` + +The `FullText` field is query-only. It is not stored with records or exposed as a selectable record property. Add another `FullText` field to create another independent index. Full-text indexes require RocksDB and an audited table. They are local derived state rebuilt from committed table data rather than authoritative record storage. See [Full-Text Search Configuration](../full-text-search/configuration.md) for the complete field contract, structured filters, phrase and prefix storage, synonyms, highlighting, and rebuild behavior. + ### `@decide` @@ -605,7 +624,7 @@ Computed properties that read other tables use the same [trusted server-side aut ### Computed Indexes -Computed properties can be indexed with `@indexed`, enabling custom indexing strategies such as composite indexes, full-text search, or vector indexing: +Computed properties can be indexed with `@indexed`, enabling custom lookup strategies such as composite keys. Use [`@fullText`](#fulltext) for native BM25-ranked text search and an HNSW `[Float]` field for vector indexing: ```graphql type Product @table { @@ -864,19 +883,22 @@ Graph navigation runs on the quantized (approximate) distances. For nearest-neig Harper supports the following field types: -| Type | Description | -| --------- | ---------------------------------------------------------------------------------------------- | -| `String` | Unicode text, UTF-8 encoded | -| `Int` | 32-bit signed integer (−2,147,483,648 to 2,147,483,647) | -| `Long` | 54-bit signed integer (−9,007,199,254,740,992 to 9,007,199,254,740,992) | -| `Float` | 64-bit double precision floating point | -| `BigInt` | Integer up to ~300 digits. Note: distinct JavaScript type; handle appropriately in custom code | -| `Boolean` | `true` or `false` | -| `ID` | String; indicates a non-human-readable identifier | -| `Any` | Any primitive, object, or array | -| `Date` | JavaScript `Date` object | -| `Bytes` | Binary data as `Buffer` or `Uint8Array` | -| `Blob` | Binary large object; designed for streaming content >20KB | +| Type | Description | +| ---------- | ---------------------------------------------------------------------------------------------- | +| `String` | Unicode text, UTF-8 encoded | +| `Int` | 32-bit signed integer (−2,147,483,648 to 2,147,483,647) | +| `Long` | 54-bit signed integer (−9,007,199,254,740,992 to 9,007,199,254,740,992) | +| `Float` | 64-bit double precision floating point | +| `BigInt` | Integer up to ~300 digits. Note: distinct JavaScript type; handle appropriately in custom code | +| `Boolean` | `true` or `false` | +| `ID` | String; indicates a non-human-readable identifier | +| `Any` | Any primitive, object, or array | +| `Date` | JavaScript `Date` object | +| `Bytes` | Binary data as `Buffer` or `Uint8Array` | +| `Blob` | Binary large object; designed for streaming content >20KB | +| `FullText` | Query-only full-text index declaration; valid only with `@fullText` and never stored | + +Added `FullText` in v5.3.0 Added `BigInt` in v4.3.0 diff --git a/reference/database/transaction.md b/reference/database/transaction.md index baba4612e..d69442286 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -134,7 +134,7 @@ Deletes transaction log entries older than the specified timestamp. Deprecated i Deletes transaction log entries older than the specified timestamp. :::warning Database-wide and irreversible -On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. +On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This purges whole log files whose entries predate the timestamp, removing [`read_audit_log`](#read_audit_log) history for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. The same history can be required to bridge a [full-text index rebuild](../full-text-search/overview.md#startup-and-recovery) from its table scan to current writes. Cleanup that removes the bridge during a rebuild can fail that attempt; stop aggressive cleanup before retrying so the next attempt can scan from a fresh committed-log boundary. ::: Parameters: diff --git a/reference/full-text-search/configuration.md b/reference/full-text-search/configuration.md new file mode 100644 index 000000000..ee9ef7cfb --- /dev/null +++ b/reference/full-text-search/configuration.md @@ -0,0 +1,191 @@ +--- +title: Configuration +--- + +# Full-Text Search Configuration + + + +Declare a nullable `FullText` field on an audited RocksDB table and attach `@fullText` to that field. The field name identifies the index. Add another `FullText` field when one table needs independent indexes for different search experiences. + +```graphql +type Product @table(database: "catalog", audit: true) { + id: ID @primaryKey + name: String + description: String + tags: [String] + category: String @indexed + price: Float @indexed + available: Boolean + catalogSearch: FullText + @fullText( + fields: [ + { name: "name", weight: 3, highlight: true } + { name: "description", weight: 1, highlight: true } + { name: "tags", weight: 1 } + ] + highlighting: { maxFragments: 2, fragmentLength: 120 } + ) +} +``` + +`catalogSearch` is a query-only declaration. It is not stored, selectable, writable, or included in record, OpenAPI, or MCP schemas. Use the field name as the condition attribute and select `$score` or `$highlights` for search metadata. + +The declaration must use nullable `FullText` exactly and carry one `@fullText` directive. Lists, non-null forms, a bare `FullText` field, and additional directives such as `@computed`, `@indexed`, or `@allow` are rejected. `@fullText` is not valid on a table type or any other field type. + +Ordinary tables inherit `logging.auditLog`, which defaults to `true`. New tables with `@fullText` instead require an explicit `@table(audit: true)`; omitting it or setting `audit: false` returns `400`, even when the global default is enabled. This table-level requirement makes transaction logging an explicit recovery prerequisite for the derived index rather than relying on the ambient default. An existing table that already has persisted audit logging can add the field without restating `audit`. A previously unaudited table can enable `audit: true` in the same schema change that adds its first full-text field; the initial build scans current records and does not require older transaction history. Disabling audit logging while a full-text field exists is rejected. + +Writing the declaration-only field or selecting it as record data returns an error, including on unsealed tables. A read-through source response containing that name fails with `502` and is not cached, indexed, or audited. For HTTP-sourced caching tables, avoid index names that collide with response wrapper fields such as `data`, `headers`, `body`, and `status`. + +## Directive options + +| Option | Default | Description | +| -------------- | ------------- | ------------------------------------------------------------------------------------- | +| `fields` | required | One or more source fields. | +| `analyzer` | `"english@2"` | Versioned text analyzer. `english@2` is the only supported value. | +| `stopWords` | `true` | Removes common English terms during analysis. | +| `positions` | `true` | Stores token positions. Required for phrase queries. | +| `surfaceTerms` | `true` | Stores unstemmed analyzed terms. Required for prefix, fuzzy-prefix, and highlighting. | +| `synonyms` | `[]` | Index-time synonym expansion rules. | +| `highlighting` | omitted | Enables highlight generation and sets fragment limits. | + +Each field entry supports: + +| Option | Default | Description | +| ----------- | -------- | ------------------------------------------------------------------------------ | +| `name` | required | Stored source field name. | +| `weight` | `1` | Positive relevance weight. Higher values contribute more to BM25 ranking. | +| `highlight` | `false` | Allows highlights from this field when index highlighting is enabled. | +| `mediaType` | omitted | Required as `"text/plain"` for a `Blob` source; invalid on other source types. | + +Source fields cannot use `@computed` or `@relationship`. Indexable source values are strings, arrays containing strings or nulls, or Blob values declared as plain text. + +## Structured filters + +Combine full-text conditions with ordinary Harper conditions using `and`. Harper checks current source records and can use existing secondary indexes to narrow candidates. Only the directive options listed above are accepted; an unknown option returns `400`. See [combining conditions](./querying.md#combining-conditions). + +## Phrase and prefix support + +`positions` and `surfaceTerms` trade index size for query features. Keep their defaults unless storage measurements show that the features are unnecessary. + +```graphql +type Article @table(audit: true) { + id: ID @primaryKey + body: String + bodySearch: FullText @fullText(fields: [{ name: "body" }], positions: false, surfaceTerms: false) +} +``` + +This smaller declaration supports `matches`, `matches_all`, and `matches_fuzzy`. It rejects phrase, prefix, fuzzy-prefix, and highlighting requests. + +## Highlighting + +Highlighting is off by default. To enable it: + +1. Set `highlight: true` on at least one source field. +2. Add `highlighting` to the directive. +3. Keep `surfaceTerms: true`. + +```graphql +articleSearch: FullText + @fullText( + fields: [{ name: "title", highlight: true }, { name: "body", highlight: true }] + highlighting: { maxFragments: 3, fragmentLength: 160 } + ) +``` + +`maxFragments` and `fragmentLength` must be positive integers. Query results contain offsets into the original field value; see [Highlights](./querying.md#highlights). + +## Synonyms + +Synonyms expand terms while records are indexed: + +```graphql +catalogSearch: FullText + @fullText( + fields: [{ name: "description" }] + synonyms: [ + { source: "sneaker", replacements: ["shoe", "trainer"] } + { source: "tv", replacements: ["television"] } + ] + ) +``` + +Rules require one source term and one or more unique replacement terms. A rule cannot replace a term with itself. + +## Multiple indexes + +Add separate `FullText` fields when separate index definitions are useful: + +```graphql +type Product @table(database: "catalog", audit: true) { + id: ID @primaryKey + name: String + tags: [String] + titleSearch: FullText @fullText(fields: [{ name: "name", weight: 2 }]) + tagSearch: FullText @fullText(fields: [{ name: "tags" }]) +} +``` + +The indexes update independently and can be queried independently. A single query cannot combine conditions from different full-text indexes. + +## Blob sources + +Only plain-text Blob fields are supported: + +```graphql +type Document @table(audit: true) { + id: ID @primaryKey + content: Blob + contentSearch: FullText @fullText(fields: [{ name: "content", mediaType: "text/plain" }]) +} +``` + +`mediaType: "text/plain"` declares the expected Blob media type. Harper does not reject the source record write, but the derived index checks a nonempty runtime `Blob.type` and excludes the record when it does not match `text/plain`. The index decodes matching Blobs as UTF-8. A Blob source is limited to 1 MiB (1,048,576 bytes). A media-type mismatch, oversized value, or invalid UTF-8 removes the whole record from the index rather than indexing only its other fields. A transient Blob read failure rolls back the accepted native batch and retries it, so index coverage does not advance past unread source data. + +## Schema changes and rebuilds + +Changes that affect indexed storage create a new local index generation and rebuild from source data. These include source membership, source order, media types, analyzer settings, stop-word handling, positions, surface terms, and synonyms. Keep the `fields` list in a stable order when its meaning has not changed. + +While a generation rebuilds, ordinary table reads and writes remain available, but full-text queries on that node return retryable `503` responses until the index is ready. + +Changing source weights or highlighting settings changes query behavior without rebuilding indexed term storage. Removing the `FullText` field retires its local files after the schema change is confirmed. Renaming the field creates a new index identity and retires the old one. + +## Inspecting an index + +`describe_table` reports `full_text_indexes`, including fields, analyzer options, supported query modes, and readiness: + +```json +{ + "operation": "describe_table", + "database": "catalog", + "table": "Product" +} +``` + +The table response includes entries shaped like this: + +```json +{ + "full_text_indexes": [ + { + "name": "catalogSearch", + "fields": [{ "name": "name", "weight": 3, "highlight": true }], + "analyzer": "english@2", + "stop_words": true, + "positions": true, + "surface_terms": true, + "synonyms": [], + "highlighting": { "maxFragments": 2, "fragmentLength": 120 }, + "query_modes": ["any", "all", "fuzzy", "phrase", "prefix", "fuzzy-prefix"], + "readiness": { "state": "ready", "owner_epoch": "4", "rebuild_attempts": 0 } + } + ] +} +``` + +`readiness.state` is `unknown`, `ready`, `needs-rebuild`, `rebuilding`, or `unavailable`. Only `ready` serves full-text queries. `unknown` is an activation or owner-election transition. `needs-rebuild` means the runtime has condemned the current generation and queued an automatic rebuild. Queries in `unknown`, `needs-rebuild`, or `rebuilding` return `503` with retryable code `INDEX_REBUILDING`. `unavailable` means activation failed or the automatic rebuild budget was exhausted; queries return a generic, non-retryable `503`. Inspect the optional `readiness.reason` and the node logs, correct the underlying storage, native-module, schema, or transaction-log problem, then reload or reapply the schema to retry activation. + +`query_modes` maps to comparators as follows: `any` → `matches`, `all` → `matches_all`, `fuzzy` → `matches_fuzzy`, `phrase` → `matches_phrase`, `prefix` → `matches_prefix`, and `fuzzy-prefix` → `matches_fuzzy_prefix`. `owner_epoch` is an opaque decimal string used to fence work between derived-index owners; clients should not parse or persist it. `rebuild_attempts` is the current automatic-attempt count. `reason` is present when readiness has an operator-actionable explanation. + +The Operations API uses snake_case for response metadata such as `stop_words`, `surface_terms`, and `owner_epoch`. The nested `highlighting` value mirrors the GraphQL option names `maxFragments` and `fragmentLength`. diff --git a/reference/full-text-search/overview.md b/reference/full-text-search/overview.md new file mode 100644 index 000000000..afef74d45 --- /dev/null +++ b/reference/full-text-search/overview.md @@ -0,0 +1,84 @@ +--- +title: Overview +--- + +# Full-Text Search + + + +Harper can build a BM25-ranked full-text index from one or more fields on a table. The index is declared in the table schema, updated from committed record changes, and queried through the same `Table.search()` and REST interfaces used for other Harper queries. + +Use full-text search when users need relevance-ranked matching across product names, descriptions, tags, articles, or other natural-language fields. It supports term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching. + +## Requirements + +- The table must use RocksDB and have [transaction logging enabled](../database/transaction.md#enabling-the-transaction-log-per-table). New schema tables must declare `@table(audit: true)`; omitting it or setting `audit: false` returns `400`. An existing table that already persisted audit logging can add a full-text field without restating the option, but disabling audit is rejected while an index is declared. +- Full-text source fields must be stored `String`, `[String]`, or `Blob` values. Blob sources must be declared as `text/plain`. +- Creating or updating an LMDB table with an `@fullText` field fails with `400`. A persisted declaration encountered while opening LMDB is ignored with a warning. Migrate the table to RocksDB before activating the index. + +See [Configuration](./configuration.md) for the complete schema contract, [Querying](./querying.md) for search examples, and [Query optimization](../resources/query-optimization.md) for performance guidance. + +## How it works + +Full-text indexes are derived local state. Harper uses Tantivy, a native text-search library, for term indexing and BM25 ranking. The table records and schema remain the source of truth. + +```mermaid +flowchart LR + W[Committed table write] --> A[Harper audit stream] + A --> F[Full-text derived-index runtime] + F --> T[Local full-text index] + T --> Q[Table.search and REST] + A --> R[Replication] + R --> D[Replica audit stream] + D --> U[Replica local full-text index] +``` + +Each node builds its own index from committed transactions. Harper does not replicate full-text index files or include them in authoritative backups. After a restore or move to a node without compatible local files, Harper rebuilds the index before serving full-text queries. + +The `@fullText` declaration must also exist on each node. A replicated component deployment or schema propagation can install it, but table-record replication alone only supplies the source records. Once the declaration is present, the node builds its own index. + +This design has several consequences: + +- A committed record can be visible before its derived full-text index has applied the same transaction. Queries use a bounded staleness policy, described in [Freshness controls](./querying.md#freshness-controls). +- A missing, corrupt, or incompatible index can be rebuilt from the table and audit stream without changing source records. +- If an index remains behind for more than 30 seconds, Harper rejects ordinary local writes to its table until the runtime catches up or changes lifecycle state. See [Write backpressure](#write-backpressure). + +## Startup and recovery + +On restart, Harper reuses compatible local index files and replays committed changes after the saved checkpoint. If those files are missing, incompatible, or corrupt, Harper rebuilds the index locally. + +A rebuild captures a fresh committed-log boundary, scans the current table, and replays changes committed after that boundary. An unreadable or corrupt transaction log can make an attempt fail. The same applies when [transaction-log cleanup](../database/transaction.md#delete_transaction_logs_before) or automatic retention removes the history needed to bridge that scan to current writes. A later attempt captures a new boundary, so stopping aggressive cleanup can allow the rebuild to succeed without restoring already-purged history. Harper retries with backoff; after the retry budget is exhausted, readiness becomes `unavailable`. Inspect `describe_table` and the node logs for the reason before retrying activation. + +Ordinary table reads and writes remain available during a rebuild. While readiness is `unknown`, `needs-rebuild`, or `rebuilding`, full-text queries return `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. If rebuild attempts are exhausted and readiness becomes `unavailable`, queries return a generic, non-retryable `503` until an operator corrects the cause and retries activation. A new replica follows the same process after the full-text declaration reaches that node. + +Use `describe_table` to inspect each index's declaration and readiness. See [Inspecting an index](./configuration.md#inspecting-an-index). + +## Write backpressure + +Harper allows a full-text index up to 30 seconds to absorb committed changes. If the runtime remains behind beyond that budget, ordinary local writes to the indexed table fail with `503`, `code: "DERIVED_INDEX_LAGGING"`, and `retryable: true`. This applies to writes through REST, the Operations API, and direct table calls from application code. Replication notifications, source applies, and crash-recovery replay continue so the authoritative data can converge. + +Retry rejected writes with backoff rather than immediately adding more load. Inspect index readiness and node logs. Write admission resumes after the runtime proves durable catch-up or moves into another lifecycle state such as rebuilding or unavailable. + +## Ranking and result loading + +The native index ranks matching documents with BM25. Harper then loads the authoritative table records for the matching IDs and discards stale or deleted candidates. Selecting `$score` includes the relevance score in each result. + +Full-text search can be combined with structured filters. Harper checks conditions against current source records before returning them. In v5.4, additional ranked batches can fill a filtered page within request budgets; exhausting a budget returns an error rather than a budget-truncated page. + +## Limits and troubleshooting + +- Phrase search requires `positions: true`; prefix and fuzzy-prefix require `surfaceTerms: true`. +- Deep offsets revisit earlier matches and increase fetched work and source reads. Prefer narrower filters and a bounded maximum offset over repeatedly raising work budgets. +- Before v5.4, prefix and fuzzy-prefix expressions restrict `offset + limit` to 100. In v5.4, bounded full-text and autocomplete queries fetch additional batches within request budgets. Use a bounded `limit` and see [paging and prefix result windows](./querying.md#paging-and-prefix-result-windows); native windows are not total-result cutoffs for bounded v5.4 queries. +- One query can use only one full-text index. An `or` group cannot mix full-text and ordinary record conditions. +- Each Blob source is limited to 1 MiB (1,048,576 bytes). A media-type mismatch, oversized value, or invalid UTF-8 removes the entire record from the index, including its other full-text source fields, until valid content is indexed. +- `INDEX_REBUILDING` is transient and retryable. `DERIVED_INDEX_LAGGING` can mean a query coverage wait expired or local writes were shed after prolonged index lag. `unavailable` means automatic rebuild attempts were exhausted or activation failed; inspect readiness and logs rather than retrying every `503` indefinitely. +- Generated GraphQL field arguments remain equality conditions; use `Table.search()`, REST, or `search_by_conditions` for full-text comparators. + +See [Querying](./querying.md) for exact condition rules and [Configuration](./configuration.md#inspecting-an-index) for readiness details. + +## Stored index data + +The local index stores tokenized terms, posting lists, optional positions and surface terms, and checkpoint metadata. It does not store an authoritative copy of each record. Harper always loads current table records before returning results, and it can discard and rebuild this derived state from committed data. + +Protect the index directory with the same filesystem controls as Harper data. Deleted source terms can remain in obsolete Tantivy segment files until normal merge and reclamation removes them; removal from query results does not imply immediate physical erasure. diff --git a/reference/full-text-search/querying.md b/reference/full-text-search/querying.md new file mode 100644 index 000000000..65cb45883 --- /dev/null +++ b/reference/full-text-search/querying.md @@ -0,0 +1,258 @@ +--- +title: Querying +--- + +# Querying Full-Text Indexes + + + +Use the query-only `FullText` field name as a condition attribute in `Table.search()`. + +```javascript +const products = Product.search({ + conditions: [{ attribute: 'catalogSearch', comparator: 'matches', value: 'waterproof trail shoes' }], + limit: 20, +}); +``` + +`Table.search()` returns an async iterable. Consume it with `for await...of`, or return it directly from a custom Resource to stream the results. + +Results are ordered by BM25 relevance. The `FullText` field, not one of its source fields, identifies the search target. + +## Match modes + +| Comparator | Behavior | Required index option | +| ---------------------- | --------------------------------------------------------------------- | --------------------- | +| `matches` | Matches any analyzed query term. | — | +| `matches_all` | Requires every analyzed query term. | — | +| `matches_phrase` | Matches analyzed terms in order as a phrase. | `positions: true` | +| `matches_prefix` | Matches records whose analyzed terms start with the final query term. | `surfaceTerms: true` | +| `matches_fuzzy` | Matches terms within one edit, including a transposition. | — | +| `matches_fuzzy_prefix` | Combines one-edit fuzzy matching with a final-term prefix. | `surfaceTerms: true` | + +Every comparator also has a negated form: `not_matches`, `not_matches_all`, `not_matches_phrase`, `not_matches_prefix`, `not_matches_fuzzy`, and `not_matches_fuzzy_prefix`. + +An empty string is invalid and returns `400`. Whitespace-only text or text reduced to no terms by the analyzer, such as a query containing only enabled stop words, returns an exact empty result. + +A query containing a negated full-text condition must also contain a non-negated full-text condition on the same index. A structured record condition alone does not satisfy this requirement. The positive full-text condition bounds the candidate set; the negated condition filters it. + +```javascript +const products = Product.search({ + operator: 'and', + conditions: [ + { attribute: 'catalogSearch', comparator: 'matches', value: 'waterproof' }, + { attribute: 'catalogSearch', comparator: 'not_matches', value: 'leather' }, + ], +}); +``` + +## Restricting source fields + +By default, a condition searches every source field in the index. Use `fields` to restrict it: + +```javascript +const products = Product.search({ + conditions: [ + { + attribute: 'catalogSearch', + comparator: 'matches_phrase', + value: 'trail running', + fields: ['name', 'description'], + }, + ], +}); +``` + +Every requested field must belong to the index and be readable by the caller. Without `fields`, the caller must be allowed to read every source field in the index. Harper rejects an external REST or Operations API query with `403` if any searched source field is not readable. `$highlights` never bypasses this check. + +Direct calls through `tables` or `databases` run in a trusted server-side context. From an authenticated custom resource, pass the `context` received by the static resource method to `Table.search()` and set `checkPermission: true` in the search options to enforce the caller's table and source-field permissions. Context propagation alone does not request the table permission check. + +## Score and ordering + +Select `$score` to include the BM25 score: + +```javascript +const products = Product.search({ + conditions: [{ attribute: 'catalogSearch', comparator: 'matches', value: 'trail shoes' }], + select: ['id', 'name', '$score'], + limit: 20, +}); +``` + +Full-text results use descending relevance order. The only explicit full-text sort is `$score` descending; other sort keys and reverse iteration are rejected because they would discard the native ranking. + +Full-text queries cannot provide an exact total after authorization, structured filtering, and current-record validation. On a REST interface with `exactCount` enabled, `Prefer: count=exact` reports an unavailable total; otherwise it is downgraded to an estimated count. See [pagination and total count](../rest/querying.md#pagination-and-total-count). `Table.getRecordCount()` counts table records, not matches for a full-text query. + +## Highlights + +Highlighting returns source-field fragments and matching character spans. The index must configure highlighting and mark at least one source with `highlight: true`. A query can also search sources without that option; those sources can affect matching and ranking but are omitted from `$highlights`. + +Selecting `$highlights` turns highlighting on for the query: + +```javascript +const products = Product.search({ + conditions: [{ attribute: 'catalogSearch', comparator: 'matches_phrase', value: 'trail running' }], + select: ['id', 'name', '$score', '$highlights'], +}); +``` + +You can also set `includeHighlights: true` on the condition. A result has this shape: + +```json +{ + "id": "shoe-1", + "name": "Waterproof trail running shoe", + "$score": 2.41, + "$highlights": { + "name": [ + { + "valueIndex": 0, + "spans": [{ "start": 11, "end": 24 }], + "fragments": [ + { + "text": "Waterproof trail running shoe", + "start": 0, + "spans": [{ "start": 11, "end": 24 }] + } + ] + } + ] + } +} +``` + +Offsets are UTF-16 half-open ranges. Value-level spans address the original source value. A fragment's `start` addresses that value, while spans inside the fragment address its returned `text`. Harper returns offsets rather than HTML so the caller controls rendering and escaping. + +## Combining conditions + +Full-text conditions can be combined with structured filters using `and`: + +```javascript +const products = Product.search({ + operator: 'and', + conditions: [ + { attribute: 'catalogSearch', comparator: 'matches', value: 'trail shoe' }, + { attribute: 'category', comparator: 'equals', value: 'footwear' }, + { attribute: 'available', comparator: 'equals', value: true }, + ], + limit: 20, +}); +``` + +Harper checks the structured conditions against current source records before returning them. Existing Harper secondary indexes can narrow candidate IDs or support point checks. Rejected hits do not count toward a v5.4 page's offset and limit, but fetching them still consumes the raw-hit budget. + +An `or` group may combine conditions from the same full-text index: + +```javascript +const products = Product.search({ + operator: 'or', + conditions: [ + { attribute: 'catalogSearch', comparator: 'matches_phrase', value: 'trail running' }, + { attribute: 'catalogSearch', comparator: 'matches', value: 'hiking boot' }, + ], +}); +``` + +An `or` group cannot mix full-text and ordinary record conditions, and one query cannot combine different full-text indexes. Full-text conditions must name a field directly on the queried table; relationship and nested-property paths are not supported. Run separate queries when any of these boundaries is required. + +## Freshness controls + +A full-text index is derived from committed table changes. Each condition accepts: + +| Option | Default | Description | +| -------------------------- | ------- | ---------------------------------------------------------------------------------- | +| `maxIndexLagMilliseconds` | `3000` | Maximum accepted upper bound on index lag. Set to `0` to require current coverage. | +| `waitForIndexMilliseconds` | `0` | How long to wait for acceptable coverage, from `0` through `30000`. | + +For read-after-write behavior, require current coverage and allow a bounded wait: + +```javascript +for await (const product of Product.search({ + conditions: [ + { + attribute: 'catalogSearch', + comparator: 'matches', + value: 'new product', + maxIndexLagMilliseconds: 0, + waitForIndexMilliseconds: 10000, + }, + ], +})) { + console.log(product); +} +``` + +All full-text conditions combined into one query must use the same freshness values. Harper returns `400` when combined conditions specify different values. + +A non-waiting HTTP query can return `Harper-Index-Coverage` with the admitted state, lag upper bound, and requested tolerance. A zero-size page performs no native search and carries no coverage proof. A waiting `Table.search()` establishes coverage as its iterator is consumed. A `search_by_conditions` request completes only after its wait and search finish. Waiting queries do not emit a coverage header before completion. + +If acceptable coverage is not reached before `waitForIndexMilliseconds` expires, Harper returns `503` with `code: "DERIVED_INDEX_LAGGING"` and `retryable: true`. For `Table.search()`, the error is raised while the async iterator is consumed. An HTTP response may already be streaming, so clients should also handle a terminal stream error rather than relying only on the initial status. + +The same code is also used when prolonged derived-index lag causes Harper to reject local writes. Query waits can be retried according to the caller's freshness needs; rejected writes should use backoff while the index catches up. See [Write backpressure](./overview.md#write-backpressure). + +## Paging and prefix result windows + +`matches_prefix` and `matches_fuzzy_prefix` are autocomplete-style record searches. These modes return matching records, not a separate list of suggested terms. + +Before v5.4, any expression containing one of these modes uses a 100-record native result window, and `offset + limit` cannot exceed 100. An unbounded query with more than 100 matches returns `400` rather than silently truncating the result. + +### Paging in v5.4 + + + +Full-text and autocomplete searches use Harper's existing `limit` and `offset`. Bounded queries can pass the ordinary native result window, including the earlier 100-record prefix boundary. Within one request, one native reader fetches ranked batches until Harper has enough eligible records for the offset and limit, exhausts the matches, or reaches a request budget. Source-version checks and record filters run before records count toward the requested page. + +For example, `offset: 100` and `limit: 10` skip the first 100 eligible records and return up to the next 10, even for prefix matching. A short or empty page means matches were exhausted. Exhausting the fetched-hit or retained-result budget returns `400`; narrow the query, reduce offset or limit, or raise the named budget. Exhausting the execution budget returns `503`; narrow the query or adjust the execution budget rather than retrying it unchanged. These budget errors carry no readiness-specific `code` or `retryable` flag. If a response has already started streaming, a later error terminates that stream instead of changing its HTTP status. An unbounded query can still fail when it exceeds the native window or configured limits. Native response-envelope overflow can return `413`, including an oversized ordinary batch even when each hit would fit. Other native failures can return `500`. A `503` without a readiness code is not a signal to wait for an index rebuild; inspect its message and adjust the work or concurrency that caused it. + +`storage.fullText.maxRawHitsPerRequest`, `storage.fullText.maxResultsPerRequest` and `storage.fullText.requestBudgetMilliseconds` bound fetched work, retained results and execution time. Replayed or rejected hits consume raw work; a request can exhaust that budget before reaching its result target. The execution budget is not a hard response-time guarantee; reader acquisition and some native work can take additional time. See [storage configuration](../configuration/options.md#storage) for the controls and [REST paging and counts](../rest/querying.md#pagination-and-total-count) for URL syntax and count behavior. + +Each offset request runs a new query and revisits earlier matches, so deeper offsets increase fetched work and source reads. Prefer narrower filters and a bounded maximum offset over repeatedly raising budgets. Each query can see changed records, permissions or index contents. Pages can therefore repeat or omit records during concurrent changes. There are no continuation tokens, close operations or retained searches across requests. + +## REST + +Use the comparator in the collection query string: + +```http +GET /Product/?catalogSearch=matches=waterproof%20trail&limit(20) +GET /Product/?catalogSearch=matches_phrase=trail%20running&select(id,name,$score,$highlights) +GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&limit(10) +``` + +REST supports all positive and negated full-text comparators. It can express the query text, selected fields, and pagination. + +Repeat the index parameter to combine the required positive and negated conditions: + +```http +GET /Product/?catalogSearch=matches=waterproof&catalogSearch=not_matches=leather&limit(20) +``` + +REST can return configured highlights by selecting `$highlights`. Use `Table.search()` or `search_by_conditions` for explicit `fields`, condition-level `includeHighlights`, or freshness controls. + +## Operations API + +`search_by_conditions` uses the same condition shape: + +```json +{ + "operation": "search_by_conditions", + "database": "catalog", + "table": "Product", + "limit": 20, + "get_attributes": ["id", "name", "$score", "$highlights"], + "conditions": [ + { + "attribute": "catalogSearch", + "comparator": "matches_all", + "value": "waterproof trail", + "fields": ["name", "description"], + "includeHighlights": true, + "maxIndexLagMilliseconds": 0, + "waitForIndexMilliseconds": 10000 + } + ] +} +``` + +If an index is `unknown`, `needs-rebuild`, or `rebuilding`, the query returns `503` with `code: "INDEX_REBUILDING"` and `retryable: true`. An index in terminal `unavailable` state returns a generic, non-retryable `503`; inspect readiness and logs instead of retrying it as a rebuild. Other busy paths can also return a generic `503`, so clients should branch on the code rather than the status alone. Unreadable source fields return `403`. Invalid declarations, unsupported match modes, incompatible combinations, and out-of-range options return `400` responses. + +For paging-budget and native response-envelope failures, see [paging and prefix result windows](#paging-and-prefix-result-windows). diff --git a/reference/graphql-querying/overview.md b/reference/graphql-querying/overview.md index fb56ffc31..79b5f6c68 100644 --- a/reference/graphql-querying/overview.md +++ b/reference/graphql-querying/overview.md @@ -172,6 +172,8 @@ fragment sharedFields on Any { Any attribute can be used as an argument for a query. In this short form, multiple arguments is treated as multiple equivalency conditions with the default `and` operation. +These arguments always use equality. A query-only `FullText` declaration field is omitted from the generated GraphQL record schema (Added in: v5.3.0); use [full-text search through `Table.search()`, REST, or the Operations API](../full-text-search/querying.md) for match comparators and BM25 ranking. + For example, the following query requires an `id` variable to be provided, and the system will search for a `Dog` record matching that id. ```graphql diff --git a/reference/index.md b/reference/index.md index 306c06837..a5606d1b3 100644 --- a/reference/index.md +++ b/reference/index.md @@ -42,6 +42,7 @@ For concept introductions, tutorials, and guides, see the [Learn](/learn) sectio | Section | Description | | ------------------------------------------------------------ | ------------------------------------------------------------------ | | [Backups](./backups/overview.md) | Managed incremental backups, snapshot downloads, and restore | +| [Full-Text Search](./full-text-search/overview.md) | BM25-ranked native search, schema configuration, and query modes | | [Logging](./logging/overview.md) | Log configuration, the `logger` API, and log management operations | | [Analytics](./analytics/overview.md) | Resource and storage analytics, system tables | | [MQTT](./mqtt/overview.md) | MQTT broker configuration and usage | diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 21815c98a..24d71951d 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -47,7 +47,7 @@ Detailed documentation: [Database Overview](../database/overview.md) ### `describe_all` -Returns the definitions of all databases and tables within the database. Record counts above 5000 records are estimated; the response includes `estimated_record_range` when estimated. To force an exact count (requires full table scan), include `"exact_count": true`. Each table definition includes the [record-structure dictionary fields](#describe_table) described for `describe_table`. +Returns the definitions of all databases and tables within the database. Record counts above 5000 records are estimated; the response includes `estimated_record_range` when estimated. To force an exact count (requires full table scan), include `"exact_count": true`. Each table definition includes the [record-structure dictionary fields](#record-structure-dictionaries) described for `describe_table`. ```json { "operation": "describe_all" } @@ -55,7 +55,7 @@ Returns the definitions of all databases and tables within the database. Record ### `describe_database` -Returns all table definitions within the specified database. Each table definition includes the [record-structure dictionary fields](#describe_table) described for `describe_table`. +Returns all table definitions within the specified database. Each table definition includes the [record-structure dictionary fields](#record-structure-dictionaries) described for `describe_table`. ```json { "operation": "describe_database", "database": "dev" } @@ -63,12 +63,43 @@ Returns all table definitions within the specified database. Each table definiti ### `describe_table` -Returns the definition of a specific table. +Returns the definition of a specific table, including its schema and storage metadata. ```json { "operation": "describe_table", "table": "dog", "database": "dev" } ``` +#### Full-text indexes + + + +A table with native full-text indexes includes `full_text_indexes`; each entry reports its query-only field name, source fields, analyzer options, enabled query modes, and `readiness` (`unknown`, `ready`, `needs-rebuild`, `rebuilding`, or `unavailable`). The readiness object also includes `owner_epoch`, `rebuild_attempts`, and an optional `reason`. Query-only `FullText` fields do not appear in the ordinary `attributes` list. + +Requesting a table with full-text indexes: + +```json +{ "operation": "describe_table", "table": "Product", "database": "catalog" } +``` + +The response also includes the table's ordinary schema and storage metadata. The relevant full-text fragment is: + +```json +{ + "name": "Product", + "full_text_indexes": [ + { + "name": "catalogSearch", + "fields": [{ "name": "name", "weight": 3 }], + "analyzer": "english@2", + "query_modes": ["any", "all", "fuzzy", "phrase", "prefix", "fuzzy-prefix"], + "readiness": { "state": "ready", "owner_epoch": "4", "rebuild_attempts": 0 } + } + ] +} +``` + +#### Record-structure dictionaries + Alongside the schema, the response carries the size of the table's **record-structure dictionaries** @@ -284,6 +315,10 @@ Returns records matching one or more conditions. Supports `operator` (`and`/`or` } ``` +Full-text conditions (Added in: v5.3.0): + +Conditions can query a declared full-text index with the `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix` comparators and their `not_` forms. A negated full-text condition requires a non-negated condition on the same full-text index. The Operations API preserves advanced condition options such as source `fields`, `includeHighlights`, `maxIndexLagMilliseconds`, and `waitForIndexMilliseconds`. See [Querying Full-Text Indexes](../full-text-search/querying.md#operations-api). + --- ## Bulk Operations diff --git a/reference/resources/query-optimization.md b/reference/resources/query-optimization.md index dfe2cf9d1..1e2675a6e 100644 --- a/reference/resources/query-optimization.md +++ b/reference/resources/query-optimization.md @@ -48,6 +48,14 @@ When a query is executed, conditions are evaluated against the database. Indexed **Rule of thumb**: Use `equals`, range operators, and `starts_with` on indexed fields. Avoid `contains`, `ends_with`, and `not_equal` as the sole or first condition in large datasets. +### Full-text search + + + +A condition using a `FullText` field declared with `@fullText` is executed by the native full-text engine and ranked with BM25. Compatible `and` filters narrow the matches. Harper loads current source records and removes stale or deleted candidates before returning them. + +Keep result pages bounded. Before v5.4, prefix and fuzzy-prefix expressions restrict `offset + limit` to 100. In v5.4, bounded queries can traverse beyond that window within request budgets; an unbounded query can still fail when it exceeds a native window or configured limit. See [paging and prefix result windows](../full-text-search/querying.md#paging-and-prefix-result-windows). Use source-field weights to tune ranking, and enable positions, surface terms, and highlighting only when their query features are needed. See [Full-Text Search](../full-text-search/overview.md). + ### Indexed vs. Non-Indexed Fields Indexed fields provide `O(log n)` lookup — fast even as the dataset grows. Non-indexed fields require `O(n)` full table scans. diff --git a/reference/resources/resource-api.md b/reference/resources/resource-api.md index da064a259..d141d2a86 100644 --- a/reference/resources/resource-api.md +++ b/reference/resources/resource-api.md @@ -1027,13 +1027,19 @@ The `Query` object is accepted by `search()` and the static `get()` method. Array of condition objects to filter records. Each condition: -| Property | Description | -| ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `attribute` | Property name, or an array for chained/joined properties (e.g. `['brand', 'name']`) | -| `value` | The value to match | -| `comparator` | `equals` (default), `greater_than`, `greater_than_equal`, `less_than`, `less_than_equal`, `starts_with`, `contains`, `ends_with`, `between`, `not_equal` | -| `conditions` | Nested conditions array | -| `operator` | `and` (default) or `or` for the nested `conditions` | +| Property | Description | +| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `attribute` | Property name, or an array for chained/joined properties (e.g. `['brand', 'name']`) | +| `value` | The value to match | +| `comparator` | `equals` (default), `greater_than`, `greater_than_equal`, `less_than`, `less_than_equal`, `starts_with`, `contains`, `ends_with`, `between`, `not_equal`, or a [full-text comparator](../full-text-search/querying.md#match-modes) | +| `conditions` | Nested conditions array | +| `operator` | `and` (default) or `or` for the nested `conditions` | +| `fields` | Full-text source fields to search; every name must belong to the selected full-text index | +| `includeHighlights` | Include configured full-text highlights | +| `maxIndexLagMilliseconds` | Maximum accepted full-text index lag; see [freshness controls](../full-text-search/querying.md#freshness-controls) | +| `waitForIndexMilliseconds` | Time to wait for acceptable full-text coverage; see [freshness controls](../full-text-search/querying.md#freshness-controls) | + +The final four properties apply to full-text conditions (v5.3.0). See [Querying Full-Text Indexes](../full-text-search/querying.md) for their constraints, error behavior, and permission requirements for direct server-side searches. Example with nested conditions: @@ -1086,6 +1092,13 @@ Special properties: - `$updatedtime` — Returns the last-updated timestamp - `$distance` — When the query ranks or filters by a vector index, returns the computed distance from the target vector. See [Vector Indexing](../database/schema.md#vector-indexing). + + +Full-text queries add two special properties: + +- `$score`: When a query uses a full-text index, returns its BM25 relevance score. +- `$highlights`: Enables and returns full-text fragments and matching spans when the index has highlighting configured. See [Highlights](../full-text-search/querying.md#highlights). + #### Selecting related records When a field is defined as a [relationship](../database/schema.md#relationships), `select` can pull the related record(s) into each result as a nested property — the programmatic equivalent of the REST `select(name,author{name})` syntax. @@ -1154,6 +1167,12 @@ If `true`, returns conditions reordered as Harper will execute them (for debuggi If `true`, forces conditions to execute in the order supplied, disabling Harper's automatic re-ordering optimization. +### `checkPermission` + +Trusted server-side calls skip table authorization by default. When delegating a request to `Table.search()`, set `checkPermission: true` on the query and pass the authenticated `context` as the second argument. Harper then performs a one-shot read authorization using `context.user`, including table, selected-property, and full-text source-field permissions. Passing the context alone does not request this check. Record-level restrictions require a synchronous, server-controlled `rowFilter`. + +Set this flag in server-controlled code whenever the endpoint promises authorized access. Never accept its value or copy a permission object from client data. + --- ## RequestTarget @@ -1167,7 +1186,7 @@ Properties: - `id` — Primary key derived from the path - `isCollection` — `true` when the request targets a collection - `rowFilter` — Synchronous JavaScript predicate applied to candidate records during search. It cannot be set by REST or QUERY request data. -- `checkPermission` — Framework-owned, one-shot request to run legacy operation authorization. Harper arms it for permission-checked dispatches. Trusted server-side code making a subsequent direct Resource call may set it to `true`, causing built-in table gates to derive permissions from `context.user`. Never accept or copy this field or a permission object from client data. +- `checkPermission` — The same one-shot authorization request available on the [Query Object](#checkpermission). Harper arms it for permission-checked dispatches. Trusted server-side code making a direct Resource call may set it to `true`, causing built-in table gates to derive permissions from `context.user`. Never accept or copy this field or a permission object from client data. Standard `URLSearchParams` methods are available: diff --git a/reference/rest/headers.md b/reference/rest/headers.md index 71603c802..04ca435c9 100644 --- a/reference/rest/headers.md +++ b/reference/rest/headers.md @@ -22,6 +22,19 @@ These headers are included in all Harper REST API responses: Collection responses to a [count request](./querying.md#pagination-and-total-count) additionally include `Content-Range`, `Range-Unit`, and `Preference-Applied` (). +### `Harper-Index-Coverage` + + + +A non-waiting request that uses a full-text index can include `Harper-Index-Coverage`. Its value is one of: + +- `current; lag=0; tolerance=`: the index covered current committed data at admission. +- `bounded; lag=; tolerance=`: the admitted lag upper bound was within the requested tolerance. + +Multiple native searches append one value each. The header certifies coverage when the search is admitted; it does not certify a later traversal. See [Full-text freshness controls](../full-text-search/querying.md#freshness-controls). + +When CORS is enabled, Harper adds this header to `Access-Control-Expose-Headers` so browser clients can read it. + ## Cache-Control diff --git a/reference/rest/querying.md b/reference/rest/querying.md index b8d7d0e66..81d4a017b 100644 --- a/reference/rest/querying.md +++ b/reference/rest/querying.md @@ -70,6 +70,34 @@ For date fields, colons must be URL-encoded as `%3A`: GET /Product/?listDate=gt=2017-03-08T09%3A30%3A00.000Z ``` +### Full-Text Operators + + + +For a `FullText` field carrying `@fullText`, use the field name as the query attribute and choose a full-text comparator: + +```http +GET /Product/?catalogSearch=matches=waterproof%20trail +GET /Product/?catalogSearch=matches_phrase=trail%20running +GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&limit(10) +``` + +REST supports `matches`, `matches_all`, `matches_phrase`, `matches_prefix`, `matches_fuzzy`, and `matches_fuzzy_prefix`, plus a `not_` form of each. A negated full-text condition requires a non-negated condition on the same index. Repeat the parameter to provide both conditions: + +```http +GET /Product/?catalogSearch=matches=waterproof&catalogSearch=not_matches=leather +``` + +REST's `limit(start,end)` uses an exclusive end. In v5.4, this prefix query skips 100 eligible products and requests the next 10, within request budgets: + +```http +GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&limit(100,110) +``` + +Use `Table.search()` or `search_by_conditions` when conditions need source-field selection, condition-level highlighting, or freshness controls. See [Querying Full-Text Indexes](../full-text-search/querying.md) for ranking, highlights, query combinations, and freshness controls. + +The REST URL syntax searches every source field in the index, so the caller must be allowed to read every source field. Before v5.4, prefix and fuzzy-prefix queries restrict `offset + limit` to 100. In v5.4, bounded full-text and autocomplete queries can pass that native window within request budgets; source-version checks and filters run before records count toward the offset and limit. Each page is a new query and can see concurrent changes. See [paging and prefix result windows](../full-text-search/querying.md#paging-and-prefix-result-windows). Full-text queries cannot provide an exact total count. On a REST interface with `exactCount: true`, a `Prefer: count=exact` request returns an unavailable total (`Content-Range: items /*`). Without that opt-in, Harper applies `count=estimated` instead and can return an estimate. + ### Chained Conditions (Range) Omit the attribute name on the second condition to chain it against the same attribute: @@ -185,10 +213,10 @@ Counting is opt-in: without the header, no count is computed and no count header Send a `Prefer` header on a `GET` (or `HEAD`) request to a collection: -| Value | Meaning | -| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `count=exact` | The exact number of matching records. Scans the full matched set, so it is off by default (see [below](#enabling-exact-counts)) and served as an estimate unless enabled for the REST interface. | -| `count=estimated` | A fast planner/table estimate. Cheap, approximate. | +| Value | Meaning | +| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `count=exact` | The exact number of matching records when the query plan can count them. Scans the full matched set, so it is off by default (see [below](#enabling-exact-counts)) and served as an estimate unless enabled for the REST interface. When enabled, full-text and other custom-index queries report an unavailable total instead. | +| `count=estimated` | A fast planner/table estimate. Cheap, approximate. | ```http GET /Product/?category=software&limit(0,25) @@ -216,7 +244,7 @@ The response status is always `200` — `Content-Range` is informational (Harper ### Unavailable totals -The total is reported as `*` (for example `Content-Range: items 0-24/*`) when it cannot be produced — an exact scan that hits its guardrail (counting the tail past the requested page is bounded by a 1,000,000-row cap and a ~1-second budget, and abandons the total rather than truncating the page), or a query with no cardinality estimate (for example a `!=` or `=ct=` (contains) condition). `Preference-Applied` still echoes the requested mode, so an unavailable total (`.../*`) is distinct from a request that asked for no count. +The total is reported as `*` (for example `Content-Range: items 0-24/*`) when it cannot be produced — an exact scan that hits its guardrail (counting the tail past the requested page is bounded by a 1,000,000-row cap and a ~1-second budget, and abandons the total rather than truncating the page), a full-text or other custom-index query, or a query with no cardinality estimate (for example a `!=` or `=ct=` (contains) condition). `Preference-Applied` still echoes the requested mode, so an unavailable total (`.../*`) is distinct from a request that asked for no count. ### Enabling exact counts diff --git a/release-notes/v5-lincoln/5.3.md b/release-notes/v5-lincoln/5.3.md index 195c6baa9..84ba47850 100644 --- a/release-notes/v5-lincoln/5.3.md +++ b/release-notes/v5-lincoln/5.3.md @@ -8,7 +8,7 @@ Harper 5.3.0 is the first stable release of the 5.3 line. It adds native full-te These notes summarize what changed relative to 5.2. The [Harper Pro v5.3.0 release on GitHub](https://github.com/HarperFast/harper-pro/releases/tag/v5.3.0) is the complete list of changes: it includes the Harper core notes, the changes that were already backported to 5.2.x, and notes for anyone who ran a 5.3 pre-release. -### Patch Releases +## Patch Releases All patch release notes for 5.3.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.3&expanded=true). @@ -37,6 +37,8 @@ Read these before upgrading a production node or cluster. Tables can declare `@fullText` fields and query them with BM25-ranked search through the existing `Table.search`, REST, and Operations API paths, with the same authorization as other queries ([harper#2855](https://github.com/HarperFast/harper/pull/2855)). Declared fields are projected into local Tantivy indexes provided by the optional `@harperfast/fulltext` package. RocksDB and the audit log remain the source of truth: the index files can be rebuilt, are reused across restarts when compatible, and are caught up from the audit log before the index reports ready ([harper#2569](https://github.com/HarperFast/harper/pull/2569), [harper#2615](https://github.com/HarperFast/harper/pull/2615), [harper#2616](https://github.com/HarperFast/harper/pull/2616)). Dropping a database or table releases every worker's index handles, and a drop that is interrupted is retried. +Search supports any-term, all-term, phrase, prefix, fuzzy, and fuzzy-prefix matching, with optional source weights, synonyms, scores, and highlights. See [Full-Text Search](/reference/v5/full-text-search/overview) and [Add Full-Text Search to a Product Catalog](/learn/developers/full-text-search-product-catalog). + ## Vector Search: Native HNSW Index A newly declared HNSW index now uses a native, memory-mapped graph file (the optional `@harperfast/hnsw` package) instead of a graph stored in RocksDB, when it is eligible: the table is on RocksDB and has audit enabled (already recorded for the table, or set with `audit: true` in the same declaration), and the index uses the default int8-quantized cosine distance and default graph-structure options (`M`, `efConstruction`, `mL`, `optimizeRouting`). An ineligible index that leaves `nativePlane` unset, or any index that sets `nativePlane: false`, uses the previous graph. Setting `nativePlane: true` on an ineligible index is refused. Existing indexes keep the graph they were built with. Queries rerank only the rows they return, and index lag behind writes is bounded, with a way for a caller to wait until its prior writes are searchable ([harper#2430](https://github.com/HarperFast/harper/pull/2430), [harper#2657](https://github.com/HarperFast/harper/pull/2657), [harper#2658](https://github.com/HarperFast/harper/pull/2658)). diff --git a/sidebarsLearn.ts b/sidebarsLearn.ts index 721b1690c..e77f63c0d 100644 --- a/sidebarsLearn.ts +++ b/sidebarsLearn.ts @@ -66,6 +66,11 @@ const sidebarsLearn: SidebarsConfig = { id: 'developers/write-through-caching', label: 'Write-Through Caching', }, + { + type: 'doc', + id: 'developers/full-text-search-product-catalog', + label: 'Full-Text Product Search', + }, { type: 'doc', id: 'developers/deploying-from-ci', diff --git a/sidebarsReference.ts b/sidebarsReference.ts index f997c1810..8ce7f8104 100644 --- a/sidebarsReference.ts +++ b/sidebarsReference.ts @@ -89,6 +89,29 @@ const sidebars: SidebarsConfig = { }, ], }, + { + type: 'category', + label: 'Full-Text Search', + collapsible: false, + className: 'reference-category-header', + items: [ + { + type: 'doc', + id: 'full-text-search/overview', + label: 'Overview', + }, + { + type: 'doc', + id: 'full-text-search/configuration', + label: 'Configuration', + }, + { + type: 'doc', + id: 'full-text-search/querying', + label: 'Querying', + }, + ], + }, { type: 'category', label: 'AI & Models',