Repository navigation
feat(staged-query): read dotted index fields, orderBy names and lookup keys as nested paths - #87
Merged
Merged
Conversation
Add FieldPath<T>, a field of T or a dot-separated path starting at one of its fields, and use it for the otherKey of Stream.lookup and Datum.lookup. State in the TSDoc and the docs that a dotted lookup key, orderBy name, or index field is a nested path, never a literal top-level key. Drop two oxlint directives that no longer suppress anything and lower their ceiling.
An unsorted MongoDB scan already returns the rows in ascending title.fr order, so the ascending case passed on a driver that sorts on a literal key.
Upd4ting
added a commit
to AntelopeJS/postgres
that referenced
this pull request
Nov 13, 2026
…indexes (#91) ## Why `orderBy("title.fr")`, `lookup(other, localKey, "title.fr")` and an index `{ fields: ["title.fr"] }` read a top-level key literally named `title.fr` instead of the `fr` member of `title`. Localized and `_internal` sorts and lookups therefore answered wrong on postgres: every row tied in a sort, and a lookup found nothing. The mysql benchmark (AntelopeJS/mysql `bench/RESULTS.md`) marks these postgres rows ✗. ANT-2089. ## Summary A dotted name now reads a nested path, with the rule AntelopeJS/mysql uses. A field declared under the exact name wins. Otherwise the name is a path below its first segment, which may be a declared JSON field (`title.fr`) or an undeclared key (`_internal.data.title.fr`). - `NamedField(source, name)` in `row-source.ts` is the one reading. - `IndexColumns` (`orderBy`, `getAll`, `between`, `group`) and the other key of `lookup` read through it. - The index DDL builds its key with the same `NamedField` over the table's own columns. The index expression is therefore the expression the read compiles to, and the planner uses it. - Two cases PostgreSQL cannot index are handled at registration, because a failed build stops the schema's whole index phase: - A path below a declared value with no members (`name.fr` over `name: "string"`) reads through `to_jsonb`, which is not IMMUTABLE, and always reads NULL. Registration refuses it with a clear message, as mysql does. - A `date` member below a declared field reads through a `timestamptz` cast, which is not IMMUTABLE. Its index keys on the stored JSON member, with a JSON null read as absent. It builds and enforces `unique`, and serves no read. - Index signatures of non-dotted names (declared fields, `_id`, undeclared keys, GIN, `crossInstance` twins) are byte-identical to `main`, so no deployed index rebuilds except those over dotted names. - `docs/storage.md` documents the rule; `docs/errors.md` lists the new refusal and the reworded reserved-column message. - `@antelopejs/interface-database` rises to `>=0.3.3 <0.4.0`, the release of AntelopeJS/interface-database#87 that carries the shared `dotted_paths` conformance suite. ```mermaid flowchart LR N["name: 'title.fr'"] --> D{"declared under<br/>the exact name?"} D -- yes --> C["that column"] D -- no --> P["source.field('title')<br/>then member 'fr'"] C --> R["read: orderBy / getAll / between /<br/>group / lookup other key"] P --> R C --> I["index key DDL"] P --> I I --> X{"IMMUTABLE?"} X -- "path below a scalar" --> F["refused at registration"] X -- "date member" --> J["key on stored JSON member"] X -- yes --> S["same expression as the read:<br/>index serves it"] ``` ## Related issue ANT-2089. The shared contract and conformance suite come from ANT-2090 (AntelopeJS/interface-database#87). ## Verification `src/test/nested-paths.test.ts`, run with `pnpm exec ajs module test . -f dist/test/nested-paths.test.js`. Before (`origin/main` source, same test file): ``` dotted names read nested paths 1) orderBy sorts by a path below a declared field 2) orderBy sorts by a path below an undeclared key 3) orderBy by an index over a path sorts by that path 4) getAll by an index over a path and between by a path match that path 5) lookup finds the books row by a path below a declared field 6) lookup finds the books row by a path below an undeclared key 7) lookup finds the booksIndexed row by a path below a declared field 8) lookup finds the booksIndexed row by a path below an undeclared key 9) a unique index over a path refuses a second row with the same nested value 10) an index over a date below a declared field builds, refuses a duplicate, and the sort reads dates ✔ an index over a date below a declared field holds a stored null as absent ✔ the index over a path serves a sort and a match by that path 2 passing 10 failing 1) orderBy sorts by a path below a declared field: AssertionError: expected [ 'book-0', 'book-1', 'book-2', …(2) ] to deeply equal [ 'book-1', 'book-4', 'book-3', …(2) ] 4) getAll by an index over a path and between by a path match that path: AssertionError: expected [] to deeply equal [ 'book-3' ] 5) lookup finds the books row by a path below a declared field: AssertionError: expected [ undefined, undefined ] to deeply equal [ 'book-3', 'book-1' ] 9) a unique index over a path refuses a second row with the same nested value: AssertionError: the insert of book-9 was not refused ``` After: ``` dotted names read nested paths ✔ orderBy sorts by a path below a declared field ✔ orderBy sorts by a path below an undeclared key ✔ orderBy by an index over a path sorts by that path ✔ getAll by an index over a path and between by a path match that path ✔ lookup finds the books row by a path below a declared field ✔ lookup finds the books row by a path below an undeclared key ✔ lookup finds the booksIndexed row by a path below a declared field ✔ lookup finds the booksIndexed row by a path below an undeclared key ✔ a unique index over a path refuses a second row with the same nested value ✔ an index over a date below a declared field builds, refuses a duplicate, and the sort reads dates ✔ an index over a date below a declared field holds a stored null as absent ✔ the index over a path serves a sort and a match by that path 12 passing ``` The two tests that pass on `main` guard regressions of the fix rather than the original bug. The plan test fails when the reads move to the nested path and the DDL does not: the index then no longer matches the read. The null test fails when the date key drops its `NULLIF`: two rows without a date then conflict under `unique`. Both were checked by patching the built `dist`. Shared conformance suite `dotted_paths` from `@antelopejs/interface-database` 0.3.3 (`-f node_modules/@antelopejs/interface-database/dist/tests/dotted_paths.test.js`): `origin/main` 1 passing, 6 failing; this branch 7 passing, 0 failing. `src/test/unit/ddl.spec.ts` adds the refusal of a path below a scalar or a date member, and a declared dotted field (`_id.x`) keying on its own column. `pnpm run checks`: `Checks: 11 passed, 0 failed.` ## Upgrade notes for the release - An index declared over a dotted name is rebuilt on the next registration with its nested key. A `unique` one fails to rebuild if the nested values hold duplicates; the previous index keeps serving and the failure is logged. - `orderBy`, `getAll`, `between`, `group` and a `lookup` other key no longer read a top-level key literally named `a.b` from `_extra`; they read the `b` member of `a`. - A schema declaring an index on a path below a scalar field (`name.fr` over `name: "string"`) is now refused at registration. Its index never served a read. ## Checklist - [x] The pull request title follows Conventional Commits. - [x] I added or updated tests when behavior changed. - [x] I updated documentation when users or contributors are affected. - [x] I documented breaking changes and migration steps. - [x] I did not include credentials or unrelated changes.
Upd4ting
added a commit
to AntelopeJS/rethinkdb
that referenced
this pull request
Nov 13, 2026
…s as nested paths (#103) ## Why The interface-database contract from AntelopeJS/interface-database#87 makes a dotted index field, `orderBy` name, and lookup key a nested path. rethinkdb read `"title.fr"` as a literal top-level key, and `getAll` on an index with one `fields` entry threw `ReqlLogicError`. This change makes rethinkdb pass the new `dotted_paths` conformance suite. ## What changed - Each field name an index, `orderBy`, or lookup reads becomes a chain of nested reads, so `"title.fr"` reads `fr` inside `title`. - An index with one `fields` entry reads that field's value, not a one-element tuple. So `getAll`, `between`, and `lookup` on it take a scalar key, and a `multi` index of one field indexes each list element. - `@antelopejs/interface-database` moves to `>=0.3.3 <0.4.0`, the release that ships the `dotted_paths` suite. ## Scope - Owned mode only. External mode keeps its own index matching. - `distinct`, `count`, `sum`, `avg`, `min`, `max`, and `group` on a dotted field that is not an index still read a literal key. The contract does not cover them. ## Blast Radius A scoped index that is not `multi` keeps the same stored function, so most deployments need nothing. Three kinds of index get a new stored function. On its own, this PR leaves them stale, because boot creates only missing indexes. #105, stacked on this branch, makes boot rebuild them. The release ships both, with these upgrade notes, which apply once #105 is merged too: > - The first boot after the upgrade rebuilds three kinds of index whose stored function changed: an index whose name or `fields` entry is dotted, a `crossInstance` index with one `fields` entry, and a `multi` index with one `fields` entry. Each one is built under `<name>__rebuild` and swapped in when ready, so it keeps answering reads during the build. Boot waits for these builds; on a large table that takes as long as creating the index. Every other index is kept. > - During a rolling deploy, v1.8 replicas still running read the rebuilt indexes with their old keys, so their reads on these three kinds return no rows until they are replaced. > - Index names ending in `__rebuild` are now reserved, like `__i`. Rename such an index before upgrading. > - A caller that passed a one-element list as the key of a one-field index must now pass the value itself. ## Verification The conformance suite `dotted_paths.test.js` from `@antelopejs/interface-database` 0.3.3, with `pnpm exec ajs module test . -f node_modules/@antelopejs/interface-database/dist/tests/dotted_paths.test.js`. Before, on `origin/main` with the suite from a tarball of #87 at `119aca4`: ``` ✔ Insert Test Data 1) getAll on an index whose fields entry is dotted finds the row by its nested value, not a literal top-level key 2) getAll on a dotted index name without fields finds the row by its nested value, not a literal top-level key 3) orderBy on an index with a dotted field sorts by the nested value, not in insertion order 4) orderBy on a dotted name that is no index sorts by the nested value, not in insertion order 5) Stream lookup on a dotted otherKey matches the nested value, not a literal top-level key 6) Datum lookup on a dotted otherKey matches the nested value, not a literal top-level key 1 passing (36ms) 6 failing 1) Dotted Paths getAll on an index whose fields entry is dotted finds the row by its nested value, not a literal top-level key: ReqlLogicError: Expected type ARRAY but found STRING in: r.db("test-dotted-paths").table("articles").getAll(r.expr("delta").prepend("default"), { ``` After, on a fresh database, with the tarball and again with the 0.3.3 release: ``` Dotted Paths ✔ Insert Test Data ✔ getAll on an index whose fields entry is dotted finds the row by its nested value, not a literal top-level key ✔ getAll on a dotted index name without fields finds the row by its nested value, not a literal top-level key ✔ orderBy on an index with a dotted field sorts by the nested value, not in insertion order ✔ orderBy on a dotted name that is no index sorts by the nested value, not in insertion order ✔ Stream lookup on a dotted otherKey matches the nested value, not a literal top-level key ✔ Datum lookup on a dotted otherKey matches the nested value, not a literal top-level key 7 passing ``` `src/test/dotted-index-reads.test.ts` covers the paths the suite does not reach: cross-instance reads, the scan fallback, and a `multi` index of one field. Its four cases fail on `origin/main` and pass with the fix. `pnpm run checks` passes. ```mermaid flowchart LR N["field name, e.g. title.fr"] --> P["ReadPath: row('title')('fr')"] P --> I["index function (scoped __i and crossInstance)"] P --> S["cross-instance scan and hasFields"] P --> O["orderBy without an index"] P --> L["lookup otherKey match"] F["fields: one entry"] --> V["index value = that field's value, scalar key"] F2["fields: two or more"] --> T["index value = tuple, list key"] ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The contract never said whether a dotted index field,
orderByname or lookup key is a nested path, so backends disagree. Localized fields and_internalrecords from interface-database-decorators need the nested reading. This PR states that a dotted name is a nested path. It adds a conformance test that every backend runs, and it letslookup(other, local, "a.b")typecheck without a cast.Summary
flowchart LR A["dotted name 'title.fr'"] --> B{where} B -->|"IndexDefinition.fields entry, or index name without fields"| N["nested path: fr inside title"] B -->|"orderBy(name): index name, or else a field"| N B -->|"lookup otherKey: FieldPath<U>"| N N -.->|"never"| L["top-level key named 'title.fr'"]FieldPathchecks only the first segment. A strict path type would reject"title.fr"on a decorators model, which types a localized field asstringbut stores a locale record.localKeystays a top-level field, because the result type replaces that field.IndexDefinition.fieldsandStream.orderBy, and the lookup, indexes, stream and datum docs, state the rule.anti-slop/no-chained-type-assertionsceiling from 3 to 1.Evidence
The new suite is
src/tests/dotted_paths.test.ts. Each case fails on a backend that reads the literal top-level key. The rows' insertion order and theirtitle.fr,title.enandmeta.rankorders all differ.pnpm exec tsc --noEmitwithstream.tsanddatum.tsatmain:@antelopejs/postgresmain(ad431a5) and@antelopejs/rethinkdbmain(95126f3), which read the literal key:@antelopejs/mongodb1.6.0, which reads the nested path, throughpnpm exec ajs module test . -f dist/tests/dotted_paths.test.js:pnpm run checks:Checks: 11 passed, 0 failed.Scope
### Upgrade notesentry follows in thedocs(changelog)PR after release.Merge Danger
Door: two-way
Blast Radius: backends
A backend that bumps to this version and runs the conformance suite fails it until it reads nested paths. Typed callers only gain accepted keys:
FieldPath<U>containskeyof U & string, andFieldPath<any>isstring.