From 65ece530934d367e30ea99d1f149f76117191739 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 24 Oct 2026 12:16:13 +1100 Subject: [PATCH 1/2] docs(plans): the json_tree capability in the Go SDK Plans how a Go program declares, stores and queries an encrypted JSON document (EQL v4 json_tree). Most of the work is in Rust: the engine refuses the JSON index in a data plan, and the family is missing from the EQL catalog, so Go has no types to generate. Five steps, one issue each, with the decisions the Go surface needs. Refs #1347 Claude-Session: https://claude.ai/code/session_01XQN1QpaWVDGjHEg39KEJF3 --- docs/plans/2026-10-24-go-json-tree.md | 253 ++++++++++++++++++++++++++ 1 file changed, 253 insertions(+) create mode 100644 docs/plans/2026-10-24-go-json-tree.md diff --git a/docs/plans/2026-10-24-go-json-tree.md b/docs/plans/2026-10-24-go-json-tree.md new file mode 100644 index 000000000..b420edc46 --- /dev/null +++ b/docs/plans/2026-10-24-go-json-tree.md @@ -0,0 +1,253 @@ +# The json_tree capability in the Go SDK + +Status: plan, not a spec. Tracking issue: #1347. Written against the json_tree stack +(#1288 → #1325 → #1329 → #1336, with #1345 on top) and `next` (#1229). +Where this disagrees with the code, the code wins. + +## Background + +`json_tree` is the EQL v4 domain family for an encrypted JSON document. The +Rust engine (`packages/stack-encrypt/src/json/`) seals a +`serde_json::Value` as one entry per node, plus a containment projection +`q`. The SQL (`packages/eql/src/v3/json_tree/`, v4 only) answers: + +- containment `@>` (GIN over `eql_v4.json_tree_query(col)`) and `<@`; +- scalar equality at a path, sent as a containment needle; +- `->` / `->>` extraction by an encrypted path selector; +- `< <= > >=`, ranges and `ORDER BY eql_v4.ord_term(col -> sel)`, at paths + declared `ordered` (CLLW OPE); +- walks over a subtree by token (`json_tree_subtree`, `_children`), with + a fallback through the sealed root. + +The design is `docs/plans/2026-10-17-json-tree-index.md`, and the leakage +write-up is `packages/eql/docs/reference/json-tree-leakage.md`. + +Today none of this reaches Go, and that is deliberate. The design doc puts +Go out of scope (lines 336-338, 377). The pieces that exist are +placeholders that refuse: + +- `encrypt.JSON()` and `encrypt.JSONTerm` (`term.go:73-78, 157`); +- stashgen's `index=json`, which parses and is then refused + (`engine.go:265-268`); +- the `eql.SteVecDocument` and `eql.JSON` rows, marked not producible + (`eql_gen.go:61-62`). + +## Constraints + +The design principles (`docs/sdk-design-principles.md`, ADR-0007 and +ADR-0008) settle most of the shape before any Go is written. + +1. **The engine does the work (G1, ADR-0007).** Go reaches the engine only + through a data plan run in the WASI guest. The engine refuses the JSON + index in a data plan today: `IndexSpec::Json` crosses the wire and is + refused with `Reason::JsonIndexPending` (`dynamic/kind.rs:106`). The + json_tree family is also missing from the EQL catalog, from `v4::all()` + and from the target table, so `eql-codegen` emits no Go type for it. + **Most of the work is in Rust, and it comes first.** +2. **One EQL column per field (Go-11).** A json_tree field is an + `encrypt_into=` field. It binds and scans as a generated + `eql.JSONTree`, which holds the EQL JSON exactly as PostgreSQL stores + it. Go never builds or reads that JSON. +3. **Checks run at the earliest stage (G3, Go-1).** Options and ordered + paths belong in the struct tag, so they are checked at `go generate` by + asking the engine (`se_plan_check`). Go keeps no copy of the path + grammar. +4. **Named concrete types (Go-2).** Every generated query method returns a + named `eql.*Query` type, never `[]byte` or `any`. +5. **Nulls.** A null json_tree document cannot be sealed at a value's + length, because documents have no single length. So a json_tree field + needs `null=sql`, exactly as text does. stashgen's `nullMode` already + gives this answer once the type reports that it is not fixed-width. + +## The proposed Go surface + +### Declaration + +```go +type Patient struct { + _ struct{} `stash:"context=patients"` + ID int64 `stash:"id,passthrough"` + Profile json.RawMessage `stash:"profile,encrypt_into=JSONTree(ordered=$.age|$.visits[*].date)"` + Notes *Notes `stash:"notes,encrypt_into=JSONTree,null=sql"` +} +``` + +- `encrypt_into=JSONTree` takes the engine's `JsonOptions`: + - `ordered=` is a `|`-separated list of paths; + - `sealed_root=false` opts out of the sealed root, which defaults to on, + as in the engine. + + stashgen hands these to the engine unparsed, as the wire form of the + `json` index (`{"json":{"sealed_root":…,"ordered":[…]}}`). The engine + parses and checks them, so a bad path or a position in an ordered path + (`JsonOrderingAtPosition`) fails `go generate`. +- **The field's Go type** (decision 1) is `json.RawMessage`, `[]byte`, or + any type `encoding/json` marshals (a struct, `map[string]any`, `any`). +- **The value crosses the ABI as JSON text** (decision 2): a new record + kind `json` whose source value is UTF-8 JSON. The engine parses it with + `serde_json`, so number canonicalisation (`2 == 2.0`) and key handling + happen in Rust, identically for every language (G1). + + On decrypt the engine returns JSON text, and the generated code + unmarshals it into the field's type. `encoding/json` is already how + opaque structs cross (`gensupport/codec.go:345`), so this adds no new + reflection to the SDK. + +### Writes and reads + +These are unchanged in shape. `patients.Encrypt(ctx, cipher, []Patient)` +returns `[]EncryptedPatient`, with `Profile eql.JSONTree`; +`patients.Decrypt` reverses it. The field's own +`Fields.Profile.Encrypt(ctx, c, v)` returns `eql.JSONTree`. + +### Queries + +These are generated per field from what the engine reports +(`se_operators`), like every other query method: + +| Generated method | Operand | Returns | SQL | +|---|---|---|---| +| `Contains(ctx, c, needle)` | a JSON document (decision 3) | `eql.JSONTreeQuery` | `profile @> $1::eql_v4.query_json_tree` | +| `ContainedBy(ctx, c, doc)` | a JSON document | `eql.JSONTreeQuery` | `profile <@ $1::eql_v4.query_json_tree` | +| `EqAt(ctx, c, path, v)` | `string` path, scalar `v` | `eql.JSONTreeQuery` | `profile @> $1::eql_v4.query_json_tree` | +| `At.Age.Lt/Lte/Gt/Gte(ctx, c, v)` | the scalar | `eql.JSONTreeLeafQuery` | `(profile -> $2::text) < $1::eql_v4.query_json_tree_leaf` | +| `At.Age.Between(ctx, c, lo, hi)` | two scalars | `encrypt.Range[eql.JSONTreeLeafQuery]` | two predicates | +| `At.Age.Selector(ctx, c)` | none | `eql.JSONTreeSelector` (text) | `profile -> $1::text`, `ORDER BY eql_v4.ord_term(profile -> $1::text)` | +| `Selector(ctx, c, path)` | `string` path | `eql.JSONTreeSelector` | `->` / `->>` on any path | + +Ordered paths get a generated accessor under `At` (decision 4), named from +the path (`$.age` becomes `Age`, or the name in an `as=`), so a typo in an +ordered path fails to compile. Equality at a path and extraction take a +runtime path, because they work at any path. The engine refuses a bad path +with `ErrTerm` and its reason, and never echoes the path text. + +Refusals come from the engine, with its reasons: + +- `JsonValueNotScalar`, for equality with a container; +- `JsonLeafNotOrdered`, for an ordering value that is not a number or + string; +- `JsonPathNotOrdered`; +- `JsonTooDeep`, for nesting deeper than 127. + +### Extraction and walks (phase 5, decision 5) + +- **Decrypting an extracted node.** A `profile -> sel` result scans into + `eql.JSONTreeEntry`. `Fields.Profile.OpenEntry(ctx, c, []eql.JSONTreeEntry)` + returns the scalars as JSON text, and a container is refused, as the + engine refuses it (`OpenedType`). +- **Subtree walks.** `Fields.Profile.SubtreeKey(ctx, c, path)` returns + `eql.JSONTreeSubtreeKey`, which binds as the token and gates for + `json_tree_subtree(_at)`. `key.Open(ctx, c, rows)` then rebuilds the + value, including the fallback through the sealed root. + + This needs a new guest entry point, because the key's path is not on the + wire and the open has to see it. It is the least settled part, so it + ships last. + +## Work, in PR-sized steps + +Each step is one PR closing one issue. Steps 1-3 are Rust and unblock +everything after them. + +1. **Catalog and Go types (eql-bindings, eql-codegen), #1348.** + - Add the json_tree family to the catalog, to `v4::all()` and to the + target table. `JSONTree`, `JSONTreeQuery`, `JSONTreeLeafQuery` and + `JSONTreeEntry` then generate into `eql_gen.go`, with a not-producible + reason that points at step 2. `JSONTreeSelector` is a text type, so + check whether the generator can emit it or it needs hand-writing in + `encrypt/eql`. + - Report the family as not fixed-width, so stashgen requires + `null=sql`. + - Gates: `types:check`, `go_eql_parity`, the codegen tests. +2. **The data-plan form of the JSON index (stack-encrypt `dynamic`), #1349.** + - Admit `IndexSpec::Json` with its options. + - Accept the `json` source kind as JSON text, seal it to a + `JsonDocument` through the same `Layout` the typed path uses, and open + it back to JSON text. + - Teach the guest's `EqlTargets` the family. `Producible` flips to true. + - Tests: + - a dynamic-versus-typed parity test, so the same document gives the + same ciphertext under the same key and the same terms; + - the mutants gate on the diff. +3. **Query by operator for json_tree (stack-encrypt, guest), #1350.** + - Add `contains` and `contained_by`. + - Add the path operators, taking a `{path, value}` input: + `eq_at`, `lt/lte/gt/gte/between` at a path, and `selector`. + - `se_operators` reports them per field, from the field's options. + - `se_query_term` returns the query twin's JSON and domain, as for other + EQL types. + - Tests: Rust parity between the dynamic and typed paths. +4. **Go: stashgen, gensupport, the checker and the record model, #1351.** + - Add the `JSONTree(...)` options on `encrypt_into`, a `json` `Kind`, + the operator names in `parseOperatorName`, and `json` in + `parseOperatorIndex`. + - Generate `Contains`, `ContainedBy`, `EqAt`, `Selector` and the `At.*` + accessors. + - Retire the `encrypt.JSON()` / `JSONTerm` / `index=json` placeholders, + which were the SteVec shape, or repoint them, so that a single + `json` index name means json_tree. This needs a decision: see Open + questions. + - Tests: + - stashgen golden output; + - refusal tests (bad path, `null=sealed`, a position in an ordered + path); + - a fixture test that the Go-sealed document and terms equal the + Rust-sealed ones under the deterministic guest; + - a live `eql_postgres_test.go` case: round trip, `@>`, `<@`, + equality at a path, ordering and `ORDER BY` at an ordered path, and + `->`. + - Docs: the Go README, the stashgen README, `doc.go` usage, the + CHANGELOG. +5. **Extraction and walks in Go, #1352.** `OpenEntry`, `SubtreeKey` and `Open`, + behind a new guest export. Live tests for walks, the sealed-root + fallback and a node that is not a scalar. + +The semantic oracle stays in Rust: #1345 holds encrypted containment to +`jsonb`. The Go tests check wiring and byte parity, not containment +semantics again. + +## Decisions needed + +1. **Which Go types a json_tree field may have.** Recommended: + `json.RawMessage`, `[]byte`, and anything `encoding/json` marshals. + - Alternative: `json.RawMessage` only. It is the strictest option and + involves no reflection, but users would unmarshal by hand. +2. **How the value crosses the ABI.** Recommended: JSON text, so the + engine parses it and the bytes match every language. + - Alternative: a vcvalue map/sequence tree. vcffi can carry one, but + Go's `encoding/json` decodes numbers to `float64`, which loses + integers above 2^53 before the engine sees them. +3. **The type of a containment needle.** Recommended: `any` marshalled + with `encoding/json`, documented. A needle built from the field's + struct type carries zero values (`"age":0`), which silently narrows the + match. + - Alternative: the field's type `T`, which is concrete (Go-2) but has + that trap whenever `T` is a struct without `omitempty`. +4. **How ordered paths are named.** Recommended: a generated accessor per + ordered path under `At`, so a typo fails to compile. + - Alternative: a runtime path string for everything, with fewer + generated methods and errors found at run time. +5. **Scope of the first release.** Recommended: steps 1-4, which cover + write, read, containment, equality at a path, ordering and + `->` selectors. Walks and opening entries (step 5) follow. + +## Open questions + +- **`index=json` and `encrypt.JSON()`.** They were reserved for the EQL v3 + SteVec index, which is also `"json"` in `checker.go:179`. Is json_tree + the only JSON index Go will ever offer? If so, delete the placeholders. + If not, give json_tree its own name. +- **"Query building in Go"** is listed as not yet decided in the + principles (line 213). The `At.*` accessors and the runtime-path methods + are a first answer, and it should be recorded there. +- **The design doc's open items reach Go:** + - The client does not yet check that a returned entry is the one at the + path asked for (DOC:495). + - The walk budget's default (DOC:391). + - The stored entry order (DOC:494). + + None blocks steps 1-4. Step 5 should wait for the selector check. +- **Ordering.** The json_tree stack is 101 commits behind `next`, which + carries the Go `null=` work, `eql_postgres_test.go` and the + `eql-postgres` CI job. Step 4 needs both, so the stack lands on `next` + first, or step 4 rebases onto it. From 42c38f4e79e40606bd7ba39564de726a0b9757db Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 22:55:58 +1100 Subject: [PATCH 2/2] docs: revise Go JSON dynamic contract plan --- docs/plans/2026-10-24-go-json-tree.md | 629 +++++++++++------- .../2026-10-25-go-json-engine-contract.md | 206 ++++++ 2 files changed, 586 insertions(+), 249 deletions(-) create mode 100644 docs/plans/2026-10-25-go-json-engine-contract.md diff --git a/docs/plans/2026-10-24-go-json-tree.md b/docs/plans/2026-10-24-go-json-tree.md index b420edc46..38f127a0a 100644 --- a/docs/plans/2026-10-24-go-json-tree.md +++ b/docs/plans/2026-10-24-go-json-tree.md @@ -1,253 +1,384 @@ -# The json_tree capability in the Go SDK - -Status: plan, not a spec. Tracking issue: #1347. Written against the json_tree stack -(#1288 → #1325 → #1329 → #1336, with #1345 on top) and `next` (#1229). -Where this disagrees with the code, the code wins. - -## Background - -`json_tree` is the EQL v4 domain family for an encrypted JSON document. The -Rust engine (`packages/stack-encrypt/src/json/`) seals a -`serde_json::Value` as one entry per node, plus a containment projection -`q`. The SQL (`packages/eql/src/v3/json_tree/`, v4 only) answers: - -- containment `@>` (GIN over `eql_v4.json_tree_query(col)`) and `<@`; -- scalar equality at a path, sent as a containment needle; -- `->` / `->>` extraction by an encrypted path selector; -- `< <= > >=`, ranges and `ORDER BY eql_v4.ord_term(col -> sel)`, at paths - declared `ordered` (CLLW OPE); -- walks over a subtree by token (`json_tree_subtree`, `_children`), with - a fallback through the sealed root. - -The design is `docs/plans/2026-10-17-json-tree-index.md`, and the leakage -write-up is `packages/eql/docs/reference/json-tree-leakage.md`. - -Today none of this reaches Go, and that is deliberate. The design doc puts -Go out of scope (lines 336-338, 377). The pieces that exist are -placeholders that refuse: - -- `encrypt.JSON()` and `encrypt.JSONTerm` (`term.go:73-78, 157`); -- stashgen's `index=json`, which parses and is then refused - (`engine.go:265-268`); -- the `eql.SteVecDocument` and `eql.JSON` rows, marked not producible - (`eql_gen.go:61-62`). - -## Constraints - -The design principles (`docs/sdk-design-principles.md`, ADR-0007 and -ADR-0008) settle most of the shape before any Go is written. - -1. **The engine does the work (G1, ADR-0007).** Go reaches the engine only - through a data plan run in the WASI guest. The engine refuses the JSON - index in a data plan today: `IndexSpec::Json` crosses the wire and is - refused with `Reason::JsonIndexPending` (`dynamic/kind.rs:106`). The - json_tree family is also missing from the EQL catalog, from `v4::all()` - and from the target table, so `eql-codegen` emits no Go type for it. - **Most of the work is in Rust, and it comes first.** -2. **One EQL column per field (Go-11).** A json_tree field is an - `encrypt_into=` field. It binds and scans as a generated - `eql.JSONTree`, which holds the EQL JSON exactly as PostgreSQL stores - it. Go never builds or reads that JSON. -3. **Checks run at the earliest stage (G3, Go-1).** Options and ordered - paths belong in the struct tag, so they are checked at `go generate` by - asking the engine (`se_plan_check`). Go keeps no copy of the path - grammar. -4. **Named concrete types (Go-2).** Every generated query method returns a - named `eql.*Query` type, never `[]byte` or `any`. -5. **Nulls.** A null json_tree document cannot be sealed at a value's - length, because documents have no single length. So a json_tree field - needs `null=sql`, exactly as text does. stashgen's `nullMode` already - gives this answer once the type reports that it is not fixed-width. - -## The proposed Go surface - -### Declaration +# Go JSON: revised dynamic contract and customer implementation plan + +Date: 2026-10-25. Status: implementation direction agreed in API review; +contract spellings and examples remain proposals, not implemented API. +Tracking issue: [#1347](https://github.com/cipherstash/stack/issues/1347). +This plan under [#1353](https://github.com/cipherstash/stack/pull/1353) uses +merged `next` at `ebdb5de0a12bbec06842851eece283fb7faea30b` and the +[API redesign](https://github.com/cipherstash/stack/blob/c5efc48837d3cb48bc7e168f780fd616a4f4845e/docs/plans/2026-10-24-stack-encrypt-api-plan.md) +tracked in [#1365](https://github.com/cipherstash/stack/pull/1365). +The [engine findings](2026-10-25-go-json-engine-contract.md) cite the source +for the current behavior described below. The examples are proposed contract +shapes, not runnable instructions for the current SDK. + +## Recommendation + +Deliver Go JSON through the existing Rust engine and EQL guest. Implement +configured target definitions, operation-specific results, and request-bound +reads now. These are the parts of the API redesign that affect the customer +contract. The public Rust traits, derives, method renames and crate moves +can follow without changing this contract. + +Keep the full JSON operation inventory in the first release, including +extraction, subtree reads, children and counts. Opening extracted values +is part of that release, alongside full-document writes and queries. The +merged engine already provides the main read machinery; adapting it safely is part of the work. + +Adopt the forthcoming Vitamin C release separately for opaque blobs. Its +storage profile does not replace JSON tree's custom encrypted nodes or EQL +envelope. Freeze and test those existing JSON formats independently. + +This recommendation makes no short-ngram mechanism decision. JSON's result +contract must be extensible without pretending that ngram Boolean-query +requirements have already been settled. + +## Changes from the first-cut plan + +| Original proposal | Revised requirement | +| --- | --- | +| JSON options travel as a special index configuration | A registered target owns typed options, validation and lowering; the same resolved definition serves writes, queries and reads | +| One query twin/domain per target; one or two byte terms | Describe each operation's input and output shape; containment, ordering, selector and subtree preparation have different results | +| `OpenEntry(cipher, rows)` without a request | A prepared selector/read object opens its own rows; expected path, definition and keyset come from the caller, never the database | +| Subtree token plus gates, `_subtree_at` | Current token lists, self tags, client wildcard expansion and probes; `json_tree_subtree(document, tokens, budget)` | +| Selector checks and number correctness are pending | Checks have landed; #1366 fixes #1364. Preserve their tests through the binding | +| Walks ship later | All required operations ship in the customer release | +| Nullable JSON field | SQL NULL, present JSON null and missing selection are distinct states | +| JSON implicitly means the old SteVec index | JSON is a plaintext kind; the target explicitly identifies the EQL JSON-tree representation | +| Reuse current EQL context plumbing | Current EQL targets are column-only and reject context extensions; do not advertise the redesign's row binding as implemented | + +The catalog, guest dispatch, generator and Rust parity work in #1353 remain +valid. The proposal needs revised contracts rather than a second SDK or +crypto implementation. Its five tracking issues remain useful work items. + +## 1. One configured target definition + +Make the public interoperability unit a versioned definition, lowered to +internal plans. It contains field identity, source kind, target, resolved +options and declared context roles. Bind requests to a field of that +definition. Do not pass another options object to a query or decrypt call. + +Proposed minimal JSON authoring form, for the existing column-only profile: + +```json +{ + "version": 1, + "fields": { + "profile": { + "type": "json", + "context": ["users", "profile"], + "target": { + "name": "EncryptedJson", + "options": {"sealed_root": true, "ordered": ["$.salary"]} + } + } + } +} +``` + +`EncryptedJson` is the recommended role-based target name, mapping to the +current Rust `JsonTreeDocument` and SQL `public.eql_v4_json_tree`. Go spells +the generated ciphertext type `eql.EncryptedJSON`. These names are proposals; +the existing SQL domain does not need a rename. Target identifiers are +contract identifiers, not names obtained from Rust type reflection. + +A string target remains shorthand for default options. Unknown keys, +unknown targets, unsupported versions and unsupported context capabilities +fail closed. Normalize defaults once. Resolve options into the same native +`JsonOptions` used by typed Rust, including its path validation. A new +configured definition is a new immutable instance, not mutation of one used +by an in-flight query. + +Extend discovery with option schema/profile identity and per-operation +input, result and capability metadata. The existing target descriptor's +single `query`/`query_sql_domain` pair cannot describe JSON accurately. +Generate schemas and Go types from the maintained registry where possible; +keep semantic validation in Rust. Do not add an independent Go path parser. + +The guest should cache validated definitions under their complete resolved +content and contract version. Generated field accessors share that immutable +content. Guest-local cache identifiers remain internal and can be recreated; +they must not become SQL values or customer persistence formats. + +The SDK keeps its idiomatic generated `Encrypt`, `Decrypt` and `Fields` +entry points. It does not expose a Rust plan or require a Go `.using()` +method. Both tags and the existing generator policy path lower to this same +definition. Simple target options can use the tag grammar; complex quoted +paths need a structured options form, not a naive `|` split. Validate both +through the engine at generation time. + +## 2. Explicit JSON values and three separate codecs + +Use UTF-8 JSON text for plaintext documents and JSON query inputs at the +host/guest boundary. In Go, use `json.RawMessage` as the fundamental value. +A typed struct adapter may explicitly call `encoding/json` and decode into +a declared type. Never decode first into `map[string]any` with default +`float64` numbers just to send the tree to Rust. + +Do not implicitly treat an ordinary `[]byte` as JSON: Go's JSON encoder +normally treats it as a byte string. Require an explicit `json.RawMessage` +conversion for raw JSON bytes. Containment accepts an explicit JSON needle, +not necessarily the field's full struct type, whose zero-valued fields can +accidentally narrow the query. + +Across the binary ABI, the declared `json` input is a UTF-8 text/byte payload +in the ordinary transport frame. In JSON authoring/request examples below, +`json` is a string carrying that text. Define this mapping explicitly; +a JSON document's `null` is not the frame's absent-field/null marker. + +| Boundary | Representation | +| --- | --- | +| Dynamic definitions, requests and responses | Versioned control schema carried by the existing Vitamin C transport frame | +| Searchable JSON plaintext | UTF-8 JSON parsed by Rust into the existing JSON value model | +| Stored searchable document | Existing EQL v4 JSON-tree envelope, header, encrypted nodes and search terms | +| Stored opaque whole struct | Vitamin C versioned storage blob from #520/#526, sealed as one value | + +Use direct typed EQL/engine conversion when implementing the adapter; remove +the internal serialize-to-Value/deserialize bridge without changing stored +bytes. This is distinct from parsing user JSON text at ingress. + +Pin the numeric model: signed/unsigned 64-bit integers and finite binary64 +values, not arbitrary-precision PostgreSQL numeric. Preserve integer tokens +without first casting to float. Add ingress checks for unsupported integer +literals instead of accepting a parser's lossy fallback. Fractional/exponent +numbers follow the documented binary64 interpretation; JSON text, key order +and decimal spelling are not promised to round trip byte for byte. The +existing `float_roundtrip` fix must remain enabled. Separately document that +JSON numeric ordering converts to `f64`, so large distinct integers can tie. + +Specify duplicate-member handling and invalid Unicode rather than inheriting +an accidental Go/Rust difference. Recommend rejecting duplicate object keys +and invalid Unicode at JSON ingress in Rust, before key I/O. This validation +is additional adapter work, not a claim about the current parser. A typed +Go encoder may already have transformed the caller's input; its explicit +adapter contract must describe that transformation. + +For an optional document, a nil `*json.RawMessage` is SQL NULL. A non-nil +pointer containing `null` is a present encrypted JSON null. A nonoptional +`json.RawMessage` is a required document; empty bytes are invalid JSON, not +an absent document. `null=sql` is required for nullable fields, not every +JSON field. Length-padded sealed null is not supported for this target. + +## 3. Operation-specific requests and results + +Use operator names with distinct operand schemas, all bound to the same +field definition. For example: + +```json +{"operation":"contains","field":"profile","input":{"json":"{\"active\":true}"}} +``` + +```json +{"operation":"eq_at","field":"profile","input":{"path":"$.name","json":"\"Alice\""}} +``` + +```json +{"operation":"subtree","field":"profile","input":{"path":"$.visits[*]","wildcard_bound":64}} +``` + +These fragments are operation bodies; the request envelope additionally +identifies protocol version, definition and execution/keyset inputs. + +Replace the assumption that every operation yields one or two anonymous +byte strings with a tagged result model. Predicate operands, ordered ranges, +selectors and prepared reads have separate variants and named Go types. +Each operand states its wire encoding and SQL parameter type. A selector +is text; subtree tokens are `text[]`; query envelopes are EQL JSON. A token +list is not a BETWEEN range or a generic Boolean OR expression. + +| Operation | Required Go/dynamic behavior | SQL use | +| --- | --- | --- | +| Encrypt/decrypt, including batches | Full document, both root policies, scalar/array/object/null | One `public.eql_v4_json_tree` column | +| Contains / ContainedBy | JSON needle and named containment query | `@>` / `<@` with `eql_v4.query_json_tree` | +| EqAt | Scalar JSON operand and validated selector-class path | Containment predicate; never ORE/OPE equality | +| LtAt/LteAt/GtAt/GteAt | Ordered path plus number/string | Extract entry, compare with `eql_v4.query_json_tree_leaf` | +| BetweenAt | Two ordered bounds plus selector | Inclusive lower/upper predicates | +| Selector | Prepared selection with named SQL selector | `->`, `->>`, first/all path query, path exists, sorting | +| Open selection | Original selection plus entries grouped by database document | Scalars; sealed root at `$`; ordinary container marker refused | +| Subtree | Prepared concrete or wildcard read with token list | `json_tree_subtree(document, tokens, budget)` | +| Open subtree | Original read plus grouped rows | Requested JSON only, including fallback | +| Children | Concrete node request and encrypted child entries | `json_tree_children(document, token)` | +| Open child scalars | Request-bound child validation in Rust | Additional adapter/helper work; never generic entry decryption | +| ChildCount | Concrete node token; nullable count | `json_tree_child_count(document, token)` | +| Stored-value comparisons | Same column/scope and compatible profile | Existing document containment and entry ordering operators | + +Generate ordered-path accessors from validated options, with explicit +aliases or a generation error on name collisions. Runtime path methods use +the same native validator. Reject positions for selector-class operations +including EqAt; those selectors erase `[n]` to `[*]`. Exact positional reads +use `subtree`. Wildcard equality is any-match; wildcard ordering through +`->` is first-match. Multiple independent EqAt predicates need not match the +same array element; use one containment needle for correlated conditions. + +For children, the SQL rows have depth one and are not complete subtrees. +Add a parent-bound opening helper that verifies child membership and opens +scalars; container children require a subtree read. Implement this in Rust +and reuse it from Go. Counts remain database observations: zero for a scalar +or empty container, NULL for absence or the sealed root. Neither counts nor +absence become authenticated merely because returned ciphertexts open. + +## 4. A read object retains the verification request + +Recommend an immutable host-owned read descriptor and guest rederivation. +Store the trusted definition/profile, resolved keyset identity, path, +wildcard bound and applicable scope inputs. Do not export PRF keys or +serialize a native `JsonSubtreeKey` expecting to recover its private state. +Its existing serialized token list is insufficient for opening checks. + +The SQL parameter is a separate value derived from the read. Opening sends +the original descriptor and untrusted rows to Rust, rederives the native +selector/subtree key and uses its existing verifier. Enforce definition, +keyset and scope agreement. Caller `context.Context` is passed per call, +never retained in the read. The current Go client has one guest instance; +rederivation also avoids imposing instance affinity on future execution. + +Proposed usage fragment, with database plumbing abbreviated deliberately: ```go -type Patient struct { - _ struct{} `stash:"context=patients"` - ID int64 `stash:"id,passthrough"` - Profile json.RawMessage `stash:"profile,encrypt_into=JSONTree(ordered=$.age|$.visits[*].date)"` - Notes *Notes `stash:"notes,encrypt_into=JSONTree,null=sql"` -} +read, err := users.Fields.Profile.Subtree(ctx, cipher, "$.visits[*]", + encrypt.JSONReadOptions{WildcardBound: 64}) +if err != nil { return err } + +// Bind read.Tokens() as PostgreSQL text[], using the driver's array adapter: +// SELECT e FROM eql_v4.json_tree_subtree($1::public.eql_v4_json_tree, +// $2::text[], $3::integer) AS e +// $1 is the stored document; $3 is the SQL walk budget, e.g. 64. + +selected, err := read.Open(ctx, cipher, encryptedRows) +if err != nil { return err } +// selected is an explicit list of JSON values, in document order. ``` -- `encrypt_into=JSONTree` takes the engine's `JsonOptions`: - - `ordered=` is a `|`-separated list of paths; - - `sealed_root=false` opts out of the sealed root, which defaults to on, - as in the engine. - - stashgen hands these to the engine unparsed, as the wire form of the - `json` index (`{"json":{"sealed_root":…,"ordered":[…]}}`). The engine - parses and checks them, so a bad path or a position in an ordered path - (`JsonOrderingAtPosition`) fails `go generate`. -- **The field's Go type** (decision 1) is `json.RawMessage`, `[]byte`, or - any type `encoding/json` marshals (a struct, `map[string]any`, `any`). -- **The value crosses the ABI as JSON text** (decision 2): a new record - kind `json` whose source value is UTF-8 JSON. The engine parses it with - `serde_json`, so number canonicalisation (`2 == 2.0`) and key handling - happen in Rust, identically for every language (G1). - - On decrypt the engine returns JSON text, and the generated code - unmarshals it into the field's type. `encoding/json` is already how - opaque structs cross (`gensupport/codec.go:345`), so this adds no new - reflection to the SDK. - -### Writes and reads - -These are unchanged in shape. `patients.Encrypt(ctx, cipher, []Patient)` -returns `[]EncryptedPatient`, with `Profile eql.JSONTree`; -`patients.Decrypt` reverses it. The field's own -`Fields.Profile.Encrypt(ctx, c, v)` returns `eql.JSONTree`. - -### Queries - -These are generated per field from what the engine reports -(`se_operators`), like every other query method: - -| Generated method | Operand | Returns | SQL | -|---|---|---|---| -| `Contains(ctx, c, needle)` | a JSON document (decision 3) | `eql.JSONTreeQuery` | `profile @> $1::eql_v4.query_json_tree` | -| `ContainedBy(ctx, c, doc)` | a JSON document | `eql.JSONTreeQuery` | `profile <@ $1::eql_v4.query_json_tree` | -| `EqAt(ctx, c, path, v)` | `string` path, scalar `v` | `eql.JSONTreeQuery` | `profile @> $1::eql_v4.query_json_tree` | -| `At.Age.Lt/Lte/Gt/Gte(ctx, c, v)` | the scalar | `eql.JSONTreeLeafQuery` | `(profile -> $2::text) < $1::eql_v4.query_json_tree_leaf` | -| `At.Age.Between(ctx, c, lo, hi)` | two scalars | `encrypt.Range[eql.JSONTreeLeafQuery]` | two predicates | -| `At.Age.Selector(ctx, c)` | none | `eql.JSONTreeSelector` (text) | `profile -> $1::text`, `ORDER BY eql_v4.ord_term(profile -> $1::text)` | -| `Selector(ctx, c, path)` | `string` path | `eql.JSONTreeSelector` | `->` / `->>` on any path | - -Ordered paths get a generated accessor under `At` (decision 4), named from -the path (`$.age` becomes `Age`, or the name in an `as=`), so a typo in an -ordered path fails to compile. Equality at a path and extraction take a -runtime path, because they work at any path. The engine refuses a bad path -with `ErrTerm` and its reason, and never echoes the path text. - -Refusals come from the engine, with its reasons: - -- `JsonValueNotScalar`, for equality with a container; -- `JsonLeafNotOrdered`, for an ordering value that is not a number or - string; -- `JsonPathNotOrdered`; -- `JsonTooDeep`, for nesting deeper than 127. - -### Extraction and walks (phase 5, decision 5) - -- **Decrypting an extracted node.** A `profile -> sel` result scans into - `eql.JSONTreeEntry`. `Fields.Profile.OpenEntry(ctx, c, []eql.JSONTreeEntry)` - returns the scalars as JSON text, and a container is refused, as the - engine refuses it (`OpenedType`). -- **Subtree walks.** `Fields.Profile.SubtreeKey(ctx, c, path)` returns - `eql.JSONTreeSubtreeKey`, which binds as the token and gates for - `json_tree_subtree(_at)`. `key.Open(ctx, c, rows)` then rebuilds the - value, including the fallback through the sealed root. - - This needs a new guest entry point, because the key's path is not on the - wire and the open has to see it. It is the least settled part, so it - ships last. - -## Work, in PR-sized steps - -Each step is one PR closing one issue. Steps 1-3 are Rust and unblock -everything after them. - -1. **Catalog and Go types (eql-bindings, eql-codegen), #1348.** - - Add the json_tree family to the catalog, to `v4::all()` and to the - target table. `JSONTree`, `JSONTreeQuery`, `JSONTreeLeafQuery` and - `JSONTreeEntry` then generate into `eql_gen.go`, with a not-producible - reason that points at step 2. `JSONTreeSelector` is a text type, so - check whether the generator can emit it or it needs hand-writing in - `encrypt/eql`. - - Report the family as not fixed-width, so stashgen requires - `null=sql`. - - Gates: `types:check`, `go_eql_parity`, the codegen tests. -2. **The data-plan form of the JSON index (stack-encrypt `dynamic`), #1349.** - - Admit `IndexSpec::Json` with its options. - - Accept the `json` source kind as JSON text, seal it to a - `JsonDocument` through the same `Layout` the typed path uses, and open - it back to JSON text. - - Teach the guest's `EqlTargets` the family. `Producible` flips to true. - - Tests: - - a dynamic-versus-typed parity test, so the same document gives the - same ciphertext under the same key and the same terms; - - the mutants gate on the diff. -3. **Query by operator for json_tree (stack-encrypt, guest), #1350.** - - Add `contains` and `contained_by`. - - Add the path operators, taking a `{path, value}` input: - `eq_at`, `lt/lte/gt/gte/between` at a path, and `selector`. - - `se_operators` reports them per field, from the field's options. - - `se_query_term` returns the query twin's JSON and domain, as for other - EQL types. - - Tests: Rust parity between the dynamic and typed paths. -4. **Go: stashgen, gensupport, the checker and the record model, #1351.** - - Add the `JSONTree(...)` options on `encrypt_into`, a `json` `Kind`, - the operator names in `parseOperatorName`, and `json` in - `parseOperatorIndex`. - - Generate `Contains`, `ContainedBy`, `EqAt`, `Selector` and the `At.*` - accessors. - - Retire the `encrypt.JSON()` / `JSONTerm` / `index=json` placeholders, - which were the SteVec shape, or repoint them, so that a single - `json` index name means json_tree. This needs a decision: see Open - questions. - - Tests: - - stashgen golden output; - - refusal tests (bad path, `null=sealed`, a position in an ordered - path); - - a fixture test that the Go-sealed document and terms equal the - Rust-sealed ones under the deterministic guest; - - a live `eql_postgres_test.go` case: round trip, `@>`, `<@`, - equality at a path, ordering and `ORDER BY` at an ordered path, and - `->`. - - Docs: the Go README, the stashgen README, `doc.go` usage, the - CHANGELOG. -5. **Extraction and walks in Go, #1352.** `OpenEntry`, `SubtreeKey` and `Open`, - behind a new guest export. Live tests for walks, the sealed-root - fallback and a node that is not a scalar. - -The semantic oracle stays in Rust: #1345 holds encrypted containment to -`jsonb`. The Go tests check wiring and byte parity, not containment -semantics again. - -## Decisions needed - -1. **Which Go types a json_tree field may have.** Recommended: - `json.RawMessage`, `[]byte`, and anything `encoding/json` marshals. - - Alternative: `json.RawMessage` only. It is the strictest option and - involves no reflection, but users would unmarshal by hand. -2. **How the value crosses the ABI.** Recommended: JSON text, so the - engine parses it and the bytes match every language. - - Alternative: a vcvalue map/sequence tree. vcffi can carry one, but - Go's `encoding/json` decodes numbers to `float64`, which loses - integers above 2^53 before the engine sees them. -3. **The type of a containment needle.** Recommended: `any` marshalled - with `encoding/json`, documented. A needle built from the field's - struct type carries zero values (`"age":0`), which silently narrows the - match. - - Alternative: the field's type `T`, which is concrete (Go-2) but has - that trap whenever `T` is a struct without `omitempty`. -4. **How ordered paths are named.** Recommended: a generated accessor per - ordered path under `At`, so a typo fails to compile. - - Alternative: a runtime path string for everything, with fewer - generated methods and errors found at run time. -5. **Scope of the first release.** Recommended: steps 1-4, which cover - write, read, containment, equality at a path, ordering and - `->` selectors. Walks and opening entries (step 5) follow. - -## Open questions - -- **`index=json` and `encrypt.JSON()`.** They were reserved for the EQL v3 - SteVec index, which is also `"json"` in `checker.go:179`. Is json_tree - the only JSON index Go will ever offer? If so, delete the placeholders. - If not, give json_tree its own name. -- **"Query building in Go"** is listed as not yet decided in the - principles (line 213). The `At.*` accessors and the runtime-path methods - are a first answer, and it should be recorded there. -- **The design doc's open items reach Go:** - - The client does not yet check that a returned entry is the one at the - path asked for (DOC:495). - - The walk budget's default (DOC:391). - - The stored entry order (DOC:494). - - None blocks steps 1-4. Step 5 should wait for the selector check. -- **Ordering.** The json_tree stack is 101 commits behind `next`, which - carries the Go `null=` work, `eql_postgres_test.go` and the - `eql-postgres` CI job. Step 4 needs both, so the stack lands on `next` - first, or step 4 rebases onto it. +Provide batch opening with groups keyed by the caller's database row +association. Never concatenate different documents' entries into one walk. +Go groups transport values; Rust still gathers and executes all key requests +in one logical batch. Do not promise one HTTP request for unlimited batches: +existing provider chunk limits and distinct keysets still apply. + +Use a singular result with `Found` plus raw JSON, and a plural result with a +JSON-value list. Present JSON null is `Found=true` with text `null`; no match +is `Found=false` or an empty plural list. A singular read with multiple +results fails rather than silently choosing one. SQL first-match selection +is a separately explicit operation. + +Name the two limits separately: + +- `wildcard_bound`: defaults to 64 per wildcard, with boundary probes; + total token expansion is capped at 8,192. A found boundary probe fails + with `JsonWildcardTruncated`; excessive expansion fails before key I/O. +- `walk_budget`: SQL defaults to 64; selects sealed-root fallback. Zero + disables fallback. With an unsealed root it does not cap response size. + +Always apply the original selection after sealed-root fallback. Do not +return the whole decrypted document from a subtree request. Preserve the +engine's rejection of forged, spliced and malformed returned walks. +Document the existing limits: the server can omit whole matches/probes, +and successful opening is not proof of result completeness or database row +identity. Do not expose a `complete=true` claim from this protocol. + +## 5. Context and provider boundary: freeze only implemented guarantees + +This is the main redesign issue that cannot be solved by Go wrappers. +JSON currently derives index context and ciphertext AAD from the same +context piece. The EQL layout accepts a column identifier; dynamic targets +reject extensions and context-field plans. + +Keep column identity, search scope, ciphertext binding and provider options +as distinct roles in the new contract. Validate source roles against the +trusted definition: a primary-key field cannot supply search scope; a +scope key is only used when explicitly selected. Keyset selection and +ZeroKMS lock policy remain execution inputs, not JSON target options. + +**Delivery recommendation:** implement the complete JSON operation surface +first under the existing column-and-keyset context profile. Advertise that +profile explicitly and reject additional scope/row-binding declarations +until their lowering is implemented. This does not claim the accepted +context redesign is complete or defer JSON operations. The customer has +not stated a requirement for new row-bound JSON profiles in this task. + +Before enabling row binding, split the native PRF and AEAD inputs and test +writes, all queries and all read paths together. Search uses column plus +scope; AEAD includes the additional row binding. Decrypt requests then need +the expected row values, including when SQL returns extracted entries. +Never append a row ID to today's shared context as a shortcut. + +Pin the current context derivation as part of the storage/search profile. +Introducing the separated profile must be explicit and retain the reader +and query behavior of released profiles; it must not reinterpret old data +or silently reindex it. Decide the profile identifier and its association +with a column/definition before release. Schema version, EQL envelope/header +version and Vitamin C blob version are different version boundaries. + +## 6. Specification and acceptance gates + +Publish the first supported subset of the dynamic interoperability spec +alongside this implementation: JSON Schema, normative semantics, transport +mapping and conformance vectors. Use the general target model, not a +JSON-only replacement protocol. Unsupported future features fail explicitly; +reserved fields are not accepted and ignored. + +The schema covers definitions, target options, operations, tagged results, +read descriptors, presence states and stable errors. The semantic contract +covers number normalization, path classes, contexts, limits and validation +before key I/O. Return machine-readable codes and safe locations, without +JSON values, paths or tokens in diagnostic text. Definition/field identifiers +can identify the failing operation without echoing user values. + +Customer release gates: + +1. Catalog, options and capability discovery match Rust and generated Go; + schema/semantic fixtures reject unknown options and unsupported profiles. +2. Go writes open in Rust and Rust writes open in Go, with identical search + terms under the same inputs. Use fixed randomness only in test fixtures; + cross-opening and SQL behavior are the production compatibility criteria. +3. All operation rows above run through the real guest and PostgreSQL. + Retain the engine's jsonb containment/read oracles; add Go binding coverage. +4. Test both root policies, nested arrays, quoted keys, scalar/null roots, + SQL NULL, absence, empty containers, large integers and the #1366 float. +5. Test wildcard N-1/N/N+1, nested expansion caps, budget zero/default, + fallback equivalence, malicious row substitutions, duplicates, mixed + documents, truncated subtrees and the documented omission limitations. +6. Test actual SQL driver encodings: `driver.Valuer`/`Scanner`, pgx arrays, + explicit casts, sqlc/GORM integration where the SDK promises support. +7. Freeze exact storage/header/search vectors and context derivation; + verify guest/engine/EQL bundle compatibility. Record the accepted leakage + profile. Future Rust/package changes must retain these fixtures. + +## Implementation order and issue amendments + +| Slice | Work | Existing issue to revise | +| --- | --- | --- | +| A. Contract and registry | Versioned configured targets, operation/result schemas, defaults, profile identity, catalog and generated types | #1348 plus contract portion of #1349 | +| B. Full-document path | JSON ingress/egress, typed target adapter, definition resolution, batched writes/decrypts | #1349 | +| C. Queries and reads in Rust/guest | JSON operators, selector preparation, token-list preparation, request-bound opening, child helper, stable errors | #1350 and #1352 together | +| D. Go surface | Tags/policies, checker, generated field methods, named SQL values, read objects and examples | #1351 | +| E. Customer release proof | Go/guest/PostgreSQL tests, conformance and storage vectors, supported-profile docs | #1347 acceptance checklist | + +Start with A and a thin B/C vertical slice: one configured JSON field, batch +round trip, containment and a request-bound subtree read through the guest. +That proves the difficult seams before expanding the generator. It is an +implementation milestone, not permission to release a reduced operation set. + +Coordinate opaque storage adoption with Vitamin C #520/#526 once reviewed +and released. Keep Rust facade/package refactors off this critical path. +Align tracking issue #1347 and work items #1348–#1352 with the slices above +before assigning implementation. Their external issue descriptions have not +yet been amended by this documentation change. + +## Evidence and verification performed + +Read #1353 and its six tracking issues; inspected merged engine, EQL SQL, +Go generator/runtime, guest dispatch and the API redesign documents. Read +Vitamin C #520/#526 descriptions to distinguish their storage deliverable +from Stack integration. The accompanying engine findings link immutable +source locations. Only documentation checks ran here; no new implementation, +Go/Rust build, live database test or performance claim is reported. + +Additional primary sources: + +- [SDK principles](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/docs/sdk-design-principles.md). +- [Dynamic target resolver and descriptor](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/dynamic/target.rs). +- [Current Go query response](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/languages/golang/encrypt/query.go). +- [Guest target dispatch](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/languages/golang/encrypt/guest/src/targets.rs). +- [Current Go codec and opaque JSON storage](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/languages/golang/encrypt/gensupport/codec.go). +- [Vitamin C Rust storage profile #520](https://github.com/cipherstash/vitaminc/pull/520) + and [Go storage profile #526](https://github.com/cipherstash/vitaminc/pull/526). diff --git a/docs/plans/2026-10-25-go-json-engine-contract.md b/docs/plans/2026-10-25-go-json-engine-contract.md new file mode 100644 index 000000000..82ba99c30 --- /dev/null +++ b/docs/plans/2026-10-25-go-json-engine-contract.md @@ -0,0 +1,206 @@ +# Merged JSON engine: obligations for the Go contract + +Supporting evidence for the [Go JSON implementation plan](2026-10-24-go-json-tree.md). + +Research baseline: `origin/next` at +[`ebdb5de0a12bbec06842851eece283fb7faea30b`](https://github.com/cipherstash/stack/commit/ebdb5de0a12bbec06842851eece283fb7faea30b), +after JSON-tree #1363, leakage documentation/tests #1362 and float-roundtrip +fix #1366. This is source analysis, not a rerun of those changes' tests. +The Go API recommendations below are proposals; the named Rust/SQL behavior +is implemented at that baseline. No ngram decisions are made here. + +## What the merged implementation actually provides + +| Capability | Engine / SQL contract | Go consequence | +| --- | --- | --- | +| Store and open a document | `JsonDocument`; EQL `JsonTreeDocument` adds the version and column envelope | One document field is one JSON value, including scalar roots, arrays and JSON null; do not restrict it to a Go object/map | +| Containment | `JsonQuery`, exact element-nested projection; SQL `@>` and `<@` | Keep operand direction explicit; array-of-object containment must not combine predicates from different elements | +| Equality at a path | `AtPath` constructs a scalar containment needle | Use this equality capability, never infer equality from the ordering term | +| Ordering at a path | `JsonLeafQuery`; `<`, `<=`, `>`, `>=`, inclusive `Between`; only number/string leaves at configured paths | Configuration belongs to the shared target definition used for writes and queries | +| Selector extraction | `JsonSelector`; SQL `->`, `->>`, path query, first and exists | Selector queries collapse positions to wildcards; first means first matching selector, not a named array position | +| Subtree read | `JsonSubtreeKey`; SQL `json_tree_subtree(document, tokens, budget)` | Preserve the trusted read request through to opening its rows | +| Children / count | SQL `json_tree_children(document, token)` and `json_tree_child_count(document, token)` | These take one node token, not an arbitrary wildcard token list; returned child entries are not complete subtrees | + +Sources: [engine query types and operators](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/query.rs), +[SQL functions](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/src/v3/json_tree/functions.sql), +[SQL operators](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/src/v3/json_tree/operators.sql). + +`sealed_root` defaults to true; `ordered` defaults to empty. Ordered paths +may include wildcards, but may not name a position such as `$.scores[2]`. +A query at an unordered path fails before loading keys. Ordering through a +wildcard selector compares the first matching entry; it is not an existential +or universal comparison over all elements. Ordering numbers uses `f64`, so +different large integers can tie; text ordering folds case and accents. +These are storage/search semantics, not Go conversion choices. +[Options](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/mod.rs), +[ordering derivation](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/derive.rs), +[documented ordering profile](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/docs/reference/json-tree-leakage.md). + +## Replace the old read recipe + +The revised design derives tokens directly from paths, stores a self tag +`x` on every entry, and expands wildcards on the client. A subtree request +sends an ordered token list (`{"t":["…"]}`), not a single token plus the +old gates. A concrete path sends one token; no ancestor token must be sent. +Tokens are column-wide structural capabilities, not decryption keys, and +must not appear in routine logs. +[Read-key implementation](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/query.rs). + +Two independent controls must be named separately: + +- **Wildcard bound:** default 64 elements at each wildcard. Each expansion + includes a probe for the element at the bound. The probe names that + element itself, even if the rest of the requested path does not exist + inside it. A matching probe makes opening fail with + `JsonWildcardTruncated`. A total expansion above **8,192 tokens** fails + with `JsonWildcardTooWide`; two wildcard levels send `N² + N + 1` tokens. +- **Walk budget:** SQL default 64 returned entries across the request. + Exceeding it with a sealed root returns the root entry alone. The client + opens the whole document and evaluates the same paths and probes locally. + Zero disables fallback; an unsealed root cannot fall back and returns the + whole requested walk. This budget is not a hard response-size cap. + +[Expansion and opening](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/derive.rs), +[SQL budget behavior](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/src/v3/json_tree/functions.sql). + +An honest server returns the named node at depth zero plus descendants. +Consequently `{}`, `[]`, a scalar, and an absent path remain distinct. +`DecryptInto>` gives `Some(null)` for JSON null and `None` for +absence; `Vec` gives all wildcard results in document order; singular +`Value` fails on zero or multiple results. In Go use an explicit presence +indicator for a singular read and an explicit result list for a wildcard +read. A nullable Go pointer alone must not collapse JSON null into absence. +[Result implementations](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/derive.rs). + +## Opening requires trusted request state + +An extracted `JsonNode` / `JsonTreeEntry` does **not** implement standalone +decryption. The selector opens a scalar extraction; the original subtree +key opens walk rows. The subtree key retains concrete paths, bound, probe +positions and the keyset/column machinery needed to recompute selectors; +its JSON serialization contains only the SQL tokens, so it is insufficient +to reconstruct the verification object. +[Engine wrappers](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/mod.rs), +[EQL wrappers](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs). + +The dynamic layer must retain an opaque guest handle or rederive the read +key from a trusted application request (definition, keyset, column context, +path, bound). Database rows must never supply the purported expected path +or context. A stateless rederivation avoids pinning requests to one WASM +instance, but the guest must still reconstruct the same native key and run +the existing opening checks. Do not duplicate these checks in Go. + +The checks reject wrong selectors, inconsistent token/self-tag relationships, +wrong positions or labels, duplicate entries, mixed headers, malformed walks +and incomplete returned subtrees. They do not authenticate the result set: +another document's entries with its own header can open, a field under one +array element can substitute for the same field under another in the cases +documented by the engine, and the database can omit entire requested nodes +or probes. Therefore “verified read” must not promise authenticated absence, +completeness, or row identity. Root fallback must return only the requested +result after client selection, never accidentally hand the whole plaintext +document to a field-read caller. +[Authentication limits](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/docs/reference/json-tree-leakage.md). + +Selector extraction opens scalars, or a sealed root explicitly requested at +`$`; container entries fail because their descendants hold their contents. +The children function returns immediate entries at depth one, not the +subtree result shape. The existing subtree opener cannot simply be applied +to those child rows. A binding should expose them as encrypted entries for +SQL use, or add a specifically request-bound opening operation if plaintext +child enumeration is required. A full subtree read and local child +enumeration is an alternative with different transfer/leakage costs. +[Extraction implementation](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/derive.rs), +[children SQL](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/src/v3/json_tree/functions.sql). + +Child count returns zero for scalars and empty containers, NULL for a missing +node, and NULL for the sealed root. The latter's top-level fields do not +chain from the root token. Do not turn either NULL case into zero. + +## Remaining Go/dynamic work is real + +At this source baseline, the dynamic wire grammar can parse +`{"json":{"sealed_root":true,"ordered":["$.salary"]}}`, but runtime +admission still refuses every JSON index with `json_index_pending`. +`ValueKind` admission assumes scalar indexes. The EQL JSON family is +handwritten and explicitly absent from the catalog/target table. Dynamic +query operands currently distinguish only one term from a range, and +their operator model lacks the JSON-specific path/read inputs. Merely +generating Go types does not make this capability usable. +[Dynamic index grammar](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/dynamic/term.rs), +[admission](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/dynamic/kind.rs), +[dynamic queries](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/dynamic/query.rs), +[EQL family](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs). + +Keep three contracts separate: plaintext JSON transport across the guest, +the existing EQL document/query/read wire forms, and the Vitamin C storage +profile for opaque blobs. A new storage codec for opaque user structs does +not replace `json_tree`'s node encoding or its EQL JSON envelope. The current +EQL/engine conversion still serializes through `serde_json::Value`; direct +typed conversion is a worthwhile format-preserving implementation change. +[Conversions and layouts](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs). + +## Context roles must not be conflated + +The existing `seal_pending` takes one `CallerContext`, obtains its ZeroKMS +descriptor, then uses the same context piece for both the index PRF context +and the AEAD context. The EQL layout accepts only a nonempty `Identifier` +for the column. Dynamic EQL target fields explicitly reject caller +extensions (`TargetError::Extended`) and context-field plans +(`TargetError::ContextField`). Separate row AAD and search scope are therefore +not an existing JSON capability. +[Sealing](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/json/derive.rs#L755), +[EQL layout](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs), +[dynamic restrictions](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/src/dynamic/record.rs#L73). + +Before freezing a profile that offers row binding, lower distinct column +identity, ciphertext context and search scope into the engine. Never append +a row ID to the present shared context as a shortcut: that also changes +selectors, value terms and read tokens per row. An initial column-only +profile can explicitly refuse those extensions, but must not advertise +unsupported row AAD or reinterpret the same profile later. Tenant keyset +selection is another separate input; it does not provide a replacement for +the context-role distinction. + +For the current client and any later pooled WASI execution, prefer a +host-owned immutable read descriptor +with definition/profile identity, resolved keyset identity, search-scope +inputs, path and wildcard bound. Each open call sends that trusted descriptor +plus untrusted rows, and the guest rederives `JsonSubtreeKey` before opening. +Ciphertext-only row context is supplied separately per document. This avoids +requiring a guest-local handle to survive pool scheduling. The descriptor +must never be reconstructed from database metadata; operation parameters +and column expectations remain application-owned. Reusing a definition ID +for different options/context encodings must be forbidden or versioned. + +One stale source comment in `JsonOptions::sealed_root` says there is no root +token. After #1363 the root has a token; with a sealed root it finds only +the root entry and cannot enumerate top-level members. The module-level +description, implementation and SQL establish the current behavior. + +## Numbers and release fixtures + +#1366 enables `serde_json/float_roundtrip` on Stack Encrypt and tests the +reported one-ULP failure through document, scalar extraction and subtree +opening. This fixes #1364; do not retain it as an unresolved blocker. +It does not make PostgreSQL arbitrary-precision numeric values supported or +make the `f64` ordering encoding lossless for all integers. +[Merged fix](https://github.com/cipherstash/stack/commit/ff287a32887d6a6c07c51d64e61b00deb46eed16). + +The Go contract must preserve signed/unsigned integers through 64 bits and +finite binary64 values without first decoding every JSON number as Go +`float64`. Declare the supported number model and reject unsupported values +instead of silently rounding them. Test Rust-written/Go-opened and +Go-written/Rust-opened documents, not just same-language round trips. + +Release fixtures should cover both root settings; null/absent/empty +containers; nested arrays and quoted keys; containment grouping; equality +versus lossy ordering; concrete and wildcard reads; bound minus one, bound +and bound plus one; token-cap refusal; fallback equivalence; grouped rows +from multiple database records; malicious wrong entries and dropped probes; +and PostgreSQL parameter casts. In particular `->` with an untyped literal +can select native `jsonb` behavior: examples should use typed parameters or +explicit `::text` casts. +[Existing engine tests](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/stack-encrypt/tests/json_index.rs), +[PostgreSQL oracle](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/tests/encryption/tests/json_tree.rs), +[operator resolution warning](https://github.com/cipherstash/stack/blob/ebdb5de0a12bbec06842851eece283fb7faea30b/packages/eql/src/v3/json_tree/operators.sql).