Skip to content

feat(staged-query): read dotted index fields, orderBy names and lookup keys as nested paths - #87

Merged
Upd4ting merged 3 commits into
mainfrom
feat/dotted-paths-contract
Nov 13, 2026
Merged

Upd4ting merged 3 commits into
mainfrom
feat/dotted-paths-contract

Conversation

@Upd4ting

Copy link
Copy Markdown
Member

Why

The contract never said whether a dotted index field, orderBy name or lookup key is a nested path, so backends disagree. Localized fields and _internal records from interface-database-decorators need the nested reading. This PR states that a dotted name is a nested path. It adds a conformance test that every backend runs, and it lets lookup(other, local, "a.b") typecheck without a cast.

Summary

flowchart LR
  A["dotted name 'title.fr'"] --> B{where}
  B -->|"IndexDefinition.fields entry, or index name without fields"| N["nested path: fr inside title"]
  B -->|"orderBy(name): index name, or else a field"| N
  B -->|"lookup otherKey: FieldPath<U>"| N
  N -.->|"never"| L["top-level key named 'title.fr'"]
Loading
+export type FieldPath<T> = (keyof T & string) | `${keyof T & string}.${string}`;
 lookup(right: Stream<U>, localKey: TK,
-  otherKey: keyof U & string,
+  otherKey: FieldPath<U>,
 )
  • FieldPath checks only the first segment. A strict path type would reject "title.fr" on a decorators model, which types a localized field as string but stores a locale record.
  • localKey stays a top-level field, because the result type replaces that field.
  • The TSDoc of IndexDefinition.fields and Stream.orderBy, and the lookup, indexes, stream and datum docs, state the rule.
  • Two lint directives in the touched files suppressed nothing. The suppressions check fails on such a directive in a touched file, so this PR removes both and lowers the anti-slop/no-chained-type-assertions ceiling from 3 to 1.

Evidence

The new suite is src/tests/dotted_paths.test.ts. Each case fails on a backend that reads the literal top-level key. The rows' insertion order and their title.fr, title.en and meta.rank orders all differ.

  • Before (types): pnpm exec tsc --noEmit with stream.ts and datum.ts at main:
    src/tests/dotted_paths.test.ts(148,62): error TS2345: Argument of type '"meta.sku"' is not assignable to parameter of type '"_id" | "title" | "meta"'.
    src/tests/dotted_paths.test.ts(161,37): error TS2345: Argument of type '"meta.sku"' is not assignable to parameter of type '"_id" | "title" | "meta"'.
    
    After: exit 0, no output.
  • Before (behavior): the compiled suite run against @antelopejs/postgres main (ad431a5) and @antelopejs/rethinkdb main (95126f3), which read the literal key:
    postgres:  1 passing, 6 failing  (getAll: expected [] to deeply equal [ Array(1) ]; lookup: Target cannot be null or undefined.)
    rethinkdb: 1 passing, 6 failing  (getAll dotted field: ReqlLogicError: Expected type ARRAY but found STRING)
    
    After: @antelopejs/mongodb 1.6.0, which reads the nested path, through pnpm exec ajs module test . -f dist/tests/dotted_paths.test.js:
    ✔ getAll on an index whose fields entry is dotted finds the row by its nested value, not a literal top-level key
    ✔ getAll on a dotted index name without fields finds the row by its nested value, not a literal top-level key
    ✔ orderBy on an index with a dotted field sorts by the nested value, not in insertion order
    ✔ orderBy on a dotted name that is no index sorts by the nested value, not in insertion order
    ✔ Stream lookup on a dotted otherKey matches the nested value, not a literal top-level key
    ✔ Datum lookup on a dotted otherKey matches the nested value, not a literal top-level key
    7 passing
    
  • pnpm run checks: Checks: 11 passed, 0 failed.

Scope

  • Postgres (ANT-2089) and rethinkdb fail the new suite until they read nested paths. Their fixes are separate PRs in those repos.
  • The ### Upgrade notes entry follows in the docs(changelog) PR after release.

Merge Danger

Door: two-way

Blast Radius: backends

A backend that bumps to this version and runs the conformance suite fails it until it reads nested paths. Typed callers only gain accepted keys: FieldPath<U> contains keyof U & string, and FieldPath<any> is string.

Add FieldPath<T>, a field of T or a dot-separated path starting at one of its fields, and use it for the otherKey of Stream.lookup and Datum.lookup. State in the TSDoc and the docs that a dotted lookup key, orderBy name, or index field is a nested path, never a literal top-level key. Drop two oxlint directives that no longer suppress anything and lower their ceiling.
An unsorted MongoDB scan already returns the rows in ascending title.fr order, so the ascending case passed on a driver that sorts on a literal key.
@Upd4ting
Upd4ting merged commit b70be9a into main Nov 13, 2026
8 checks passed
@Upd4ting
Upd4ting deleted the feat/dotted-paths-contract branch November 13, 2026 08:51
Upd4ting added a commit to AntelopeJS/postgres that referenced this pull request Nov 13, 2026
…indexes (#91)

## Why

`orderBy("title.fr")`, `lookup(other, localKey, "title.fr")` and an
index `{ fields: ["title.fr"] }` read a top-level key literally named
`title.fr` instead of the `fr` member of `title`. Localized and
`_internal` sorts and lookups therefore answered wrong on postgres:
every row tied in a sort, and a lookup found nothing. The mysql
benchmark (AntelopeJS/mysql `bench/RESULTS.md`) marks these postgres
rows ✗. ANT-2089.

## Summary

A dotted name now reads a nested path, with the rule AntelopeJS/mysql
uses. A field declared under the exact name wins. Otherwise the name is
a path below its first segment, which may be a declared JSON field
(`title.fr`) or an undeclared key (`_internal.data.title.fr`).

- `NamedField(source, name)` in `row-source.ts` is the one reading.
- `IndexColumns` (`orderBy`, `getAll`, `between`, `group`) and the other
key of `lookup` read through it.
- The index DDL builds its key with the same `NamedField` over the
table's own columns. The index expression is therefore the expression
the read compiles to, and the planner uses it.
- Two cases PostgreSQL cannot index are handled at registration, because
a failed build stops the schema's whole index phase:
- A path below a declared value with no members (`name.fr` over `name:
"string"`) reads through `to_jsonb`, which is not IMMUTABLE, and always
reads NULL. Registration refuses it with a clear message, as mysql does.
- A `date` member below a declared field reads through a `timestamptz`
cast, which is not IMMUTABLE. Its index keys on the stored JSON member,
with a JSON null read as absent. It builds and enforces `unique`, and
serves no read.
- Index signatures of non-dotted names (declared fields, `_id`,
undeclared keys, GIN, `crossInstance` twins) are byte-identical to
`main`, so no deployed index rebuilds except those over dotted names.
- `docs/storage.md` documents the rule; `docs/errors.md` lists the new
refusal and the reworded reserved-column message.
- `@antelopejs/interface-database` rises to `>=0.3.3 <0.4.0`, the
release of AntelopeJS/interface-database#87 that carries the shared
`dotted_paths` conformance suite.

```mermaid
flowchart LR
  N["name: 'title.fr'"] --> D{"declared under<br/>the exact name?"}
  D -- yes --> C["that column"]
  D -- no --> P["source.field('title')<br/>then member 'fr'"]
  C --> R["read: orderBy / getAll / between /<br/>group / lookup other key"]
  P --> R
  C --> I["index key DDL"]
  P --> I
  I --> X{"IMMUTABLE?"}
  X -- "path below a scalar" --> F["refused at registration"]
  X -- "date member" --> J["key on stored JSON member"]
  X -- yes --> S["same expression as the read:<br/>index serves it"]
```

## Related issue

ANT-2089. The shared contract and conformance suite come from ANT-2090
(AntelopeJS/interface-database#87).

## Verification

`src/test/nested-paths.test.ts`, run with `pnpm exec ajs module test .
-f dist/test/nested-paths.test.js`.

Before (`origin/main` source, same test file):

```
  dotted names read nested paths
    1) orderBy sorts by a path below a declared field
    2) orderBy sorts by a path below an undeclared key
    3) orderBy by an index over a path sorts by that path
    4) getAll by an index over a path and between by a path match that path
    5) lookup finds the books row by a path below a declared field
    6) lookup finds the books row by a path below an undeclared key
    7) lookup finds the booksIndexed row by a path below a declared field
    8) lookup finds the booksIndexed row by a path below an undeclared key
    9) a unique index over a path refuses a second row with the same nested value
    10) an index over a date below a declared field builds, refuses a duplicate, and the sort reads dates
    ✔ an index over a date below a declared field holds a stored null as absent
    ✔ the index over a path serves a sort and a match by that path

  2 passing
  10 failing

  1) orderBy sorts by a path below a declared field:
      AssertionError: expected [ 'book-0', 'book-1', 'book-2', …(2) ] to deeply equal [ 'book-1', 'book-4', 'book-3', …(2) ]
  4) getAll by an index over a path and between by a path match that path:
      AssertionError: expected [] to deeply equal [ 'book-3' ]
  5) lookup finds the books row by a path below a declared field:
      AssertionError: expected [ undefined, undefined ] to deeply equal [ 'book-3', 'book-1' ]
  9) a unique index over a path refuses a second row with the same nested value:
     AssertionError: the insert of book-9 was not refused
```

After:

```
  dotted names read nested paths
    ✔ orderBy sorts by a path below a declared field
    ✔ orderBy sorts by a path below an undeclared key
    ✔ orderBy by an index over a path sorts by that path
    ✔ getAll by an index over a path and between by a path match that path
    ✔ lookup finds the books row by a path below a declared field
    ✔ lookup finds the books row by a path below an undeclared key
    ✔ lookup finds the booksIndexed row by a path below a declared field
    ✔ lookup finds the booksIndexed row by a path below an undeclared key
    ✔ a unique index over a path refuses a second row with the same nested value
    ✔ an index over a date below a declared field builds, refuses a duplicate, and the sort reads dates
    ✔ an index over a date below a declared field holds a stored null as absent
    ✔ the index over a path serves a sort and a match by that path

  12 passing
```

The two tests that pass on `main` guard regressions of the fix rather
than the original bug. The plan test fails when the reads move to the
nested path and the DDL does not: the index then no longer matches the
read. The null test fails when the date key drops its `NULLIF`: two rows
without a date then conflict under `unique`. Both were checked by
patching the built `dist`.

Shared conformance suite `dotted_paths` from
`@antelopejs/interface-database` 0.3.3 (`-f
node_modules/@antelopejs/interface-database/dist/tests/dotted_paths.test.js`):
`origin/main` 1 passing, 6 failing; this branch 7 passing, 0 failing.

`src/test/unit/ddl.spec.ts` adds the refusal of a path below a scalar or
a date member, and a declared dotted field (`_id.x`) keying on its own
column.

`pnpm run checks`: `Checks: 11 passed, 0 failed.`

## Upgrade notes for the release

- An index declared over a dotted name is rebuilt on the next
registration with its nested key. A `unique` one fails to rebuild if the
nested values hold duplicates; the previous index keeps serving and the
failure is logged.
- `orderBy`, `getAll`, `between`, `group` and a `lookup` other key no
longer read a top-level key literally named `a.b` from `_extra`; they
read the `b` member of `a`.
- A schema declaring an index on a path below a scalar field (`name.fr`
over `name: "string"`) is now refused at registration. Its index never
served a read.

## Checklist

- [x] The pull request title follows Conventional Commits.
- [x] I added or updated tests when behavior changed.
- [x] I updated documentation when users or contributors are affected.
- [x] I documented breaking changes and migration steps.
- [x] I did not include credentials or unrelated changes.
Upd4ting added a commit to AntelopeJS/rethinkdb that referenced this pull request Nov 13, 2026
…s as nested paths (#103)

## Why

The interface-database contract from AntelopeJS/interface-database#87
makes a dotted index field, `orderBy` name, and lookup key a nested
path. rethinkdb read `"title.fr"` as a literal top-level key, and
`getAll` on an index with one `fields` entry threw `ReqlLogicError`.
This change makes rethinkdb pass the new `dotted_paths` conformance
suite.

## What changed

- Each field name an index, `orderBy`, or lookup reads becomes a chain
of nested reads, so `"title.fr"` reads `fr` inside `title`.
- An index with one `fields` entry reads that field's value, not a
one-element tuple. So `getAll`, `between`, and `lookup` on it take a
scalar key, and a `multi` index of one field indexes each list element.
- `@antelopejs/interface-database` moves to `>=0.3.3 <0.4.0`, the
release that ships the `dotted_paths` suite.

## Scope

- Owned mode only. External mode keeps its own index matching.
- `distinct`, `count`, `sum`, `avg`, `min`, `max`, and `group` on a
dotted field that is not an index still read a literal key. The contract
does not cover them.

## Blast Radius

A scoped index that is not `multi` keeps the same stored function, so
most deployments need nothing. Three kinds of index get a new stored
function. On its own, this PR leaves them stale, because boot creates
only missing indexes. #105, stacked on this branch, makes boot rebuild
them. The release ships both, with these upgrade notes, which apply once
#105 is merged too:

> - The first boot after the upgrade rebuilds three kinds of index whose
stored function changed: an index whose name or `fields` entry is
dotted, a `crossInstance` index with one `fields` entry, and a `multi`
index with one `fields` entry. Each one is built under `<name>__rebuild`
and swapped in when ready, so it keeps answering reads during the build.
Boot waits for these builds; on a large table that takes as long as
creating the index. Every other index is kept.
> - During a rolling deploy, v1.8 replicas still running read the
rebuilt indexes with their old keys, so their reads on these three kinds
return no rows until they are replaced.
> - Index names ending in `__rebuild` are now reserved, like `__i`.
Rename such an index before upgrading.
> - A caller that passed a one-element list as the key of a one-field
index must now pass the value itself.

## Verification

The conformance suite `dotted_paths.test.js` from
`@antelopejs/interface-database` 0.3.3, with `pnpm exec ajs module test
. -f
node_modules/@antelopejs/interface-database/dist/tests/dotted_paths.test.js`.

Before, on `origin/main` with the suite from a tarball of #87 at
`119aca4`:

```
    ✔ Insert Test Data
    1) getAll on an index whose fields entry is dotted finds the row by its nested value, not a literal top-level key
    2) getAll on a dotted index name without fields finds the row by its nested value, not a literal top-level key
    3) orderBy on an index with a dotted field sorts by the nested value, not in insertion order
    4) orderBy on a dotted name that is no index sorts by the nested value, not in insertion order
    5) Stream lookup on a dotted otherKey matches the nested value, not a literal top-level key
    6) Datum lookup on a dotted otherKey matches the nested value, not a literal top-level key
  1 passing (36ms)
  6 failing

  1) Dotted Paths
       getAll on an index whose fields entry is dotted finds the row by its nested value, not a literal top-level key:
     ReqlLogicError: Expected type ARRAY but found STRING in:
r.db("test-dotted-paths").table("articles").getAll(r.expr("delta").prepend("default"), {
```

After, on a fresh database, with the tarball and again with the 0.3.3
release:

```
  Dotted Paths
    ✔ Insert Test Data
    ✔ getAll on an index whose fields entry is dotted finds the row by its nested value, not a literal top-level key
    ✔ getAll on a dotted index name without fields finds the row by its nested value, not a literal top-level key
    ✔ orderBy on an index with a dotted field sorts by the nested value, not in insertion order
    ✔ orderBy on a dotted name that is no index sorts by the nested value, not in insertion order
    ✔ Stream lookup on a dotted otherKey matches the nested value, not a literal top-level key
    ✔ Datum lookup on a dotted otherKey matches the nested value, not a literal top-level key

  7 passing
```

`src/test/dotted-index-reads.test.ts` covers the paths the suite does
not reach: cross-instance reads, the scan fallback, and a `multi` index
of one field. Its four cases fail on `origin/main` and pass with the
fix. `pnpm run checks` passes.

```mermaid
flowchart LR
  N["field name, e.g. title.fr"] --> P["ReadPath: row('title')('fr')"]
  P --> I["index function (scoped __i and crossInstance)"]
  P --> S["cross-instance scan and hasFields"]
  P --> O["orderBy without an index"]
  P --> L["lookup otherKey match"]
  F["fields: one entry"] --> V["index value = that field's value, scalar key"]
  F2["fields: two or more"] --> T["index value = tuple, list key"]
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant