Repository navigation
docs(plans): a JSON index for Stack Encrypt as a new EQL v4 domain, with runnable SQL prototypes - #1250
docs(plans): a JSON index for Stack Encrypt as a new EQL v4 domain, with runnable SQL prototypes#1250coderdan wants to merge 11 commits into
Conversation
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
…n and exact containment Stack Encrypt declares the JSON operators (#1209) but has no JSON index, and EQL v4 has no SteVec, so the v4 JSON wire is still free to design. This plan explores three changes to the v3 SteVec taken together: node-local ciphertexts (one entry per node, no subtree duplication), a token chain that lets core SQL walk a subtree from a client-derived start token, and group tags with a GIN-prefiltered recheck that makes containment agree with jsonb @> across array elements. The three SQL files beside the plan run on a stock PostgreSQL 16 with no extension: one pins jsonb's own containment semantics, one simulates v3 and the proposed containment against that oracle (six cross-element false positives today, none under the proposal), and one prototypes the recursive token unwrap. Refs #1231 Refs #1209 Claude-Session: https://claude.ai/code/session_01HLuZxys1NDRmAkv3uhnuoJ
… HMAC as a benchmarked option Review asked whether the token derivations, which hash a secret token beside public bytes, are open to length extension. They are not: every output is truncated to 16 bytes, so the chaining state an extension resumes from is half unknown, and every derivation starts with its own domain byte, so an extended hash names a message no component consumes. Nothing in the design accepts one of these hashes as a MAC. HMAC-SHA256 keyed by the token is recorded as decision 7, to be re-evaluated against the Phase 3 benchmarks over large documents and tables rather than adopted now out of caution. Refs #1231 Claude-Session: https://claude.ai/code/session_01HLuZxys1NDRmAkv3uhnuoJ
…e v4 SteVec Decided 2026-10-18. The scheme lands as its own domain family beside eql_v4_json, with its own CHECK, operators and functions, and touches no SteVec wire or SQL. A new domain is additive and can land in any 4.x, so the work cannot block the v4 release; and a scheme per domain type is what EQL v3's design, carried into v4, exists for, so a new scheme never needs a new EQL version. The build change becomes additive as well: a v4-only source tree rendered into the v4 bundle, nothing excluded from either. Refs #1231 Claude-Session: https://claude.ai/code/session_01HLuZxys1NDRmAkv3uhnuoJ
… a column-wide token A token is a capability over structure and is deterministic per column, so a server that sees one can re-walk every row for as long as the column lives. The chain from the root hands over t($) in two legitimate queries, a root-anchored wildcard extraction and the array functions on a root array, and that token unlocks the shape of every document in the column. Moving the head of the chain down one level removes it: the root entry carries the whole document as v3's does, the top-level fields are the chain heads, and the broadest token a query can send covers one top-level field. Whole-document decryption becomes one AEAD with no walk and no index key, which is what the existing decrypt paths do. The at-rest cost is nil at depth 1, since the payload length already reveals the document's size, and the storage cost is one encrypted copy of the plaintext bytes, about 8% to 50% of the payload depending on leaf sizes. Depth 2 and beyond leak per-field sizes at rest and grow storage linearly, so the option is a boolean, on by default, recorded in the header; decision 8 leaves its name open. Refs #1231 Claude-Session: https://claude.ai/code/session_01HLuZxys1NDRmAkv3uhnuoJ
…revealed token stays useful The sealed root caps what a token unlocks; nothing in the design caps how long, because a token is a function of the column and the path and every later row is as open to it as the rows it was used on. Per-row tokens would close that and cost a round trip per row. Epochs are the middle ground: an epoch number in the header enters the head derivation, so a token revealed under one epoch opens nothing written under the next, at the cost of one token per live epoch in a query and one lookup per row in the walk. Selectors stay unepoched so containment keeps matching every row with one needle. Not in the spike; the header field is to be reserved from the first version so adding it later is a new value, not a new shape. Refs #1231 Claude-Session: https://claude.ai/code/session_01HLuZxys1NDRmAkv3uhnuoJ
…on, per-path index policy, and further leakage mechanisms Four mechanisms join the spike and four are recorded. PostgreSQL's own containment already enforces the same-element rule the plpgsql recheck re-implemented, so projecting a document into a shape nested by array element lets native @> be exact for arrays of containers while the GIN index still serves it; grouping tags are emitted for container elements only, since a scalar element cannot be grouped wrongly. Parent tags keyed by the child's ordinal stop siblings being linkable at rest for one small integer per entry. A per-path index policy with opaque subtrees stops the column emitting value selectors and ordering terms that nobody queries, which is where v3's frequency and order leakage comes from. Length padding, ORE leaf terms, salted selectors and on-disk measurement under TOAST compression are recorded with their costs so none is rediscovered as free. Phases 1 to 3 and the decisions list pick them up. Refs #1231 Claude-Session: https://claude.ai/code/session_01HLuZxys1NDRmAkv3uhnuoJ
b1c1f53 to
677ff67
Compare
|
Rebased and re-signed. The six commits are the same content, now GPG-signed and rebased onto |
Mutation testing (cargo-mutants,
|
| caught | missed | unviable | timeout |
|---|---|---|---|
| 425 | 0 | 342 | 0 |
Every mutant in the changed lines was caught by a test.
The work this document planned is no longer a spike: the index, the EQL v4 `json_tree` domain family and the live oracle are built in the pull requests stacked on this one, and the document carries their implementation notes. The file and its prototype directory are renamed from `2026-10-17-json-index-stack-encrypt-spike` to `2026-10-17-json-tree-index`, the title and status describe it as the design and implementation record of the `json_tree` index, and "the spike" becomes "the phased plan" wherever it named the Phase 0 to 4 plan. No technical content changes. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
b8d1c87 to
55bbd66
Compare
coderdan
left a comment
coderdan
left a comment
There was a problem hiding this comment.
This is the design record for the EQL v4 json_tree type: a way to store an encrypted JSON document in PostgreSQL so the server can answer containment (@>), path extraction and subtree reads without the decryption key, plus three SQL prototypes that are meant to back its claims. The design holds together, but the containment prototype does not compute what the document says it shows. The token chain also leaks object key names to anyone holding a token, and the document says a token reveals nothing about content.
Some terms used below. A selector s is a keyed hash of a node's JSON path. A value selector vs is a keyed hash of a path and a value. A token is a secret 16-byte value per node that a query hands the server so it can walk the subtree under that node. The index key is the per-keyset secret the client derives selectors and the first tokens from. I ran all three SQL files against PostgreSQL 17.10 in a scratch database.
Findings, most severe first
-
High.
containment-oracle.sqlnever simulates containment, so the results the document reports from it were not produced by this file.docs/plans/2026-10-17-json-tree-index/containment-oracle.sql:51-52and:104-105.sim.flat_containsandsim.exact_containstake the plaintext document, but pass it straight tosim.to_queryandgroups_ok, which expect the flattened formsim.store(doc)(an object with ansvarray).doc->'sv'is NULL for every case, soflat_todayandexact_newarefalseon all 31 rows, andexact_newdisagrees withjsonb @>on 18 of them. The six v3 false positives that the Summary (line 17), the table at lines 149-156 and the "Result" paragraph (line 178) report never appear. Withsim.store(...)applied, the file gives the result the document describes: the v3 form disagrees on 8 rows (the six cross-element cases, the root-scalar case, and the2/2.0artefact), and the grouped form disagrees only on the two listed artefacts. This matters because the PR's verification section and #1231's diagnosis cite this file as evidence, and anyone who reruns it gets the opposite answer. Fix: flatten the stored side inside both functions. Add a summary query that counts disagreements per form, so the file checks itself. Say "agrees on 29 of 31; the other two are the listed artefacts" rather than "all 31". -
High. A token lets the server confirm guesses of the object key names under it, but the document says a token reveals nothing about content.
docs/plans/2026-10-17-json-tree-index.md:111, and the "key names" row of the leakage table at line 259. Below a chain head,t(child) = H("t" ‖ t(parent) ‖ 0x01 ‖ key)uses no key. A server that holdst(parent)(from a query, or because the walk has unwrapped it) can test any candidate name offline:H("t" ‖ t(parent) ‖ 0x01 ‖ guess) == t(child). The walk hands it every descendant's token, so this works at every level of the subtree. Tokens are the same in every row of the column, so one subtree query becomes a permanent dictionary oracle over the key names under that path, in every row. Key names are often data: maps keyed by email address, user ID or SKU. Selectors do not have this problem, because they are keyed with the index key. Fix: state the leak in section 2 and in the leakage table. Record the remedy as a decision: derive the label step under the index key,t(child) = PRF(json/token ‖ column, t(parent) ‖ 0x01 ‖ key). In this design the server never derives a child token from a label (it unwrapsw), so keying the step costs the server nothing. (Phase 3b in #1329 later removes the label from the token. Its object-member rankH("o" ‖ t(parent) ‖ key)has the same unkeyed shape, so the same remedy applies there.) -
Medium. The token-chain SQL, both the excerpt and the prototype file, does not implement the derivations the document specifies, and the length-extension argument depends on the difference.
docs/plans/2026-10-17-json-tree-index.md:91-105andtoken-chain.sql:3-6, 30-31, 36, 40, 62, 66. The spec (lines 82-86) puts a domain byte first and salts with the document IV:p = H("p" ‖ iv ‖ t(parent)), andt(child) = H("t" ‖ t(parent) ‖ 0x01 ‖ key). The excerpt, introduced as "fromtoken-chain.sql, which runs as written", computesh16(t ‖ 'p' ‖ iv). The file computesh16(t ‖ 'p')with no IV, and builds child tokens ash16(t ‖ '.' ‖ key), with no"t"domain and no kind byte. The SHA-256 paragraph (line 109) argues from "each derivation starts with its own domain byte", which neither SQL form does. The per-document salt that the "at rest" paragraph (line 113) relies on is absent from the prototype. Fix: bringtoken-chain.sqlto the specified derivations (domain first, the IV inpand the mask, the kind byte,LE64(index)), and quote the excerpt from the file verbatim. -
Medium.
ContainedBy(<@) has no design: no needle shape, and no index.docs/plans/2026-10-17-json-tree-index.md:50("Containment stays a GIN probe") and section 3, which describe@>only.JsonQueryis listed as answeringContainedBy(line 297), and the SQL list (line 307) names "@>and<@as the two-stage expression". Butjsonb_path_ops(the GIN operator class this design uses) supports@>,@?and@@, not<@. Socol <@ needleis evaluated row by row, with no index, as it already is in v3. The root-scalaralttrick does not carry over either, because a stored document has noalt. Fix: add a paragraph to section 3: the stored side is the needle,<@is the needle's projection containing the row's, it is a filter with no index, and the root-scalar case is one-sided. Narrow the Speed goal toContains. -
Low. The nonce is described as unique "by construction", but a truncated hash is unique only with high probability.
docs/plans/2026-10-17-json-tree-index.md:240, against the goal at line 49.H("nonce" ‖ s ‖ LE64(j))[..12]maps distinct inputs to 96 bits, so two entries collide with probability aboutN² / 2^97. That is negligible, but it is not structural. Structural uniqueness would use the ordinaljitself as a 96-bit counter nonce, which is safe here because every document gets a fresh data key. Fix: say which of the two the design means. -
Low. The sealed root's at-rest cost is described as nil, but it is not.
docs/plans/2026-10-17-json-tree-index.md:131-133. Chain heads carry nopand now(line 126), so a stored row shows which entries are top-level fields and how many there are, with no token. #1336's leakage page says this; the design does not. The storage estimate on line 133 also compares plaintext bytes with encoded characters: the copy is encoded too, so at hex it is about twice the 8% and 50% quoted. Fix: state the top-level count as the at-rest cost, and put the estimate in one unit. -
Low. Two references to the repository are wrong.
docs/plans/2026-10-17-json-tree-index.md:309namesmise run test:stack_encrypt_pg. No task by that name exists, either at the cited commit3a216441(which has only the database-freetest:encryption) or onnext. The suite the plan means istest:encryption:postgres, onnextnow. Line 325 (Phase 2) puts the SQL atsrc/v4/json/*"as listed", but the list at line 307 sayssrc/v4/json_tree/. Fix: name the real task and the listed path.
Not reported
- The ADR-0010 known non-findings. The tree structure (
p,w, the group tags, the stored projection) is not authenticated, and which entries a row holds, and in what order, is not bound. So the server can move an entry between array elements that share a selector, or drop one, without the decryption failing. That is the documented design, not a defect. - Places where the body describes a stored form that later PRs in the stack changed (
vs,n,g,wand the ordinal on each entry, the derived nonce,groups_ok, hex encoding, the size estimates). #1288, #1325 and #1329 record each change in dated notes and implementation sections, so those are not repeated here. - The oracle tags scalar array elements with
n, where the design tags container elements only. This does not change any answer. - Prose style.
… simulates sim.flat_contains and sim.exact_contains passed the plaintext stored document straight to sim.to_query and sim.groups_ok, which read the flattened form's `sv` array. `stored->'sv'` was NULL for every case, so both simulated forms returned false on all 31 rows and the grouped form disagreed with jsonb @> on 18 of them. The six v3 cross-element false positives the plan reports never appeared, so the file did not back the result the plan cites from it. Both functions now flatten the stored side with sim.store, and a closing query counts each form's disagreements with jsonb @>: 8 for the v3 form (the six cross-element cases, the root-scalar false negative, the 2 / 2.0 artefact) and 2 for the grouped form (the two listed artefacts). Rerun on PostgreSQL 17.10. The plan now says 29 of 31 rather than "every case". Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…n specifies The prototype hashed the token first and the domain byte after it, left the document IV out of the parent tag and the mask, and built child tokens as h16(t || '.' || key) with no "t" domain and no kind byte. The plan's excerpt, introduced as running "as written" from the file, was a third variant. The plan's SHA-256 paragraph argues from each derivation starting with its own domain byte, and its at-rest paragraph from the IV salt, so the prototype did not exercise either property the text relies on. token-chain.sql now derives t(child), p and the mask exactly as the plan's formulas say (domain first, the IV in p and the mask, 0x01 key / 0x02 LE64 index), keys a document by a stored IV, and adds a check that the same document under a second IV shares no parent tag. The excerpt is quoted from the file. Rerun on PostgreSQL 17.10: six entries under $.nested, none from a wrong token, all nine non-root entries from the root token, zero shared tags. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
… guarantees Review of the design found five places where the text claims more than the design delivers or names things that do not exist: - A token was said to reveal nothing about content. Below a token each child token is an unkeyed hash of the parent's token and the label, and the walk hands the server every descendant's token, so a token holder can test guesses of the key names beneath it in every row. The token section and the leakage table now say so, and decision 11 records the remedy: derive the label step under the index key, which costs the server nothing because it unwraps child tokens rather than deriving them. - ContainedBy had no design. The section on containment now says what its needle is, that jsonb_path_ops cannot index <@ so it is a per-row filter as in v3, and that the root-scalar alt is one-sided; the speed goal is narrowed to Contains. - The nonce was called unique by construction. A 96-bit truncated hash is unique with overwhelming probability; the ordinal as a counter nonce is the construction that meets the goal, and is safe because the key is per document. - The sealed root's at-rest cost was called nil, but heads without p or w show which entries are top-level fields. Its storage estimate compared plaintext bytes with encoded characters; both are now in hex characters. - The test task was test:stack_encrypt_pg, which does not exist; it is test:encryption:postgres. Phase 2 named src/v4/json where the list names src/v4/json_tree. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The json_tree stack targets next, but this branch forked from main and never took next, so every PR above it showed next's own changes (block ORE, text search terms, typed queries) as its diff. Bringing next in here shrinks #1288's view from 187 files to the 46 it changes. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…-ordinal-chain Carries origin/next up the stack from #1250's branch through #1288's and #1325's, so this PR reads against one merge base. Content-empty: this branch already had next. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Carries origin/next up from #1250 through #1329, so this PR reads against one merge base. Content-empty. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5dc913642c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "v": 4, | ||
| "k": "sv", | ||
| "i": { "t": "users", "c": "profile" }, | ||
| "h": "…", // key header, once: iv, key tag, descriptor, keyset id (opaque to SQL) |
There was a problem hiding this comment.
This wire shape leaves the per-document IV only inside h, while explicitly declaring h opaque to SQL (matching packages/eql/src/v3/json/types.sql:92-106). The server-side walk must compute p and unwrap w using that IV, and the prototype only works because chain.store adds a separate top-level iv field that is absent here. Consequently, a real subtree(doc, token) cannot evaluate the parent-tag join for any row; add a SQL-readable salt/IV field to the domain or redesign the derivations so the walk does not need data hidden in h.
Useful? React with 👍 / 👎.
|
|
||
| Wildcard paths. A client cannot derive `t($.a[*].b)`, because the token is positional. Extraction under a wildcard anchors at the nearest non-wildcard ancestor and gates on the selector: `subtree_at(doc, t($.a), s($.a[*].b))` walks from `$.a`, and emits an entry only once the walk has passed through an entry whose `s` matches. Scalar leaves under a wildcard need no token at all: `->` by selector returns them as today, and their nonce and AAD depend on `s` and `j`, not on `t`. | ||
|
|
||
| Arrays. `jsonb_array_length(doc, t)` is the count of children of `t`; `jsonb_array_elements(doc, t)` is the children, each a subtree root. Positional access (`$.a[2]`) is a token, not a selector, so the item (`[@]`) and positional (`[n]`) `ArrayIndexMode`s have no reason to exist in v4: wildcard only, and the chain for positions. |
There was a problem hiding this comment.
For positional access such as $.a[2], the supplied value is t($.a[2]), but the recursive query uses a token only to find entries whose p was derived from it—i.e. the selected node's children. A scalar element has no children and no n tag, while its wildcard selector is shared by every sibling, so this access returns no way to identify or decrypt element 2. Carry the parent token plus ordinal, or store an own-token lookup tag for every node, before retiring positional selectors.
Useful? React with 👍 / 👎.
Summary
The design record of the EQL v4
json_treeindex: searchable encrypted JSON for Stack Encrypt (our Rust encryption engine), shipped as a new domain type in EQL v4 (the SQL library we install into a customer's PostgreSQL database so it can search encrypted values). It changes three things about the v3 SteVec scheme: each entry encrypts only its own node instead of its whole subtree; a token chain lets core PostgreSQL walk a subtree from one client-derived start token, so node-local ciphertexts stay extractable in one round trip; and containment agrees with PostgreSQL's ownjsonb @>across array elements, which closes the v4 half of #1231. Three SQL prototypes beside the plan run on a stock PostgreSQL 16 with no extension and were run as part of writing it.Documentation only. No code, no changeset, nothing shipped changes.
Changes
docs/plans/2026-10-17-json-tree-index.md: where we are (cipherstash-client 0.42.3's SteVec read from source, what Stack Encrypt has onnext, what EQL v4 has), goals and constraints, the design (node-local entries, the token chain with the sealed root on by default, exact containment, wire format, keys and nonces, a leakage table against v3, performance, standard-Postgres compatibility), further mechanisms, where it lands, a five-phase plan, and ten decisions with the ones taken on 2026-10-18 recorded inline.docs/plans/2026-10-17-json-tree-index/jsonb-semantics.sql: pins PostgreSQL's own containment answers for 23 edge cases.…/containment-oracle.sql: a plaintext-selector simulation of the v3 flattening and of the proposed grouped containment, compared againstjsonb @>.…/token-chain.sql: the recursive token unwrap and subtree walk in core SQL.Verification
jsonb-semantics.sqlconfirms every semantic claim the plan makes aboutjsonb @>.containment-oracle.sqlshows v3's selector containment disagreeing withjsonb @>on six cross-element cases and the grouped form agreeing on all 31, with two simulation artefacts listed in the file.token-chain.sqlreturns exactly the six entries under$.nestedfrom a token derived on the client from the path alone, zero entries from a wrong token, and all nine non-root entries from the root token.origin/nextat3a216441.Related
Refs #1249 (the issue this plan is for)
Refs #1231, #1060, #1209, #976
Review notes
Start with the Summary and the "Decisions to take before Phase 1" list at the end; the body is the argument for each. The SHA-256 versus HMAC paragraph under "The token chain" and the "Epochs" subsection record reasoning that came out of review and are deliberately kept in the document rather than in the PR thread.
Generated by Claude Code