Repository navigation
Conversation
…ed PostgreSQL, and document what the tree reveals Two of the three items that gate a release of EQL v4's json_tree. The planner. `eql_v4.json_tree_query` is plpgsql with `COST 1000` so that a bitmap scan over the GIN index beats a sequential scan even when the documents are TOASTed and the heap is a few pages. The benches showed that on one PostgreSQL version and nothing held it: the oracle test checks the plan with sequential scans turned off, which is not what a customer's database runs. The new test seals a flat table (1,000 four-node documents, every row inline) and a nested one (300 documents of about 220 nodes, every row TOASTed), pins both regimes by `pg_column_size`, runs `ANALYZE`, and with every setting at its default asserts a bitmap scan over the GIN index for containment and a scan of the functional btree for a range at an ordered path. A control prices the function at 1 for one transaction and shows the sequential scan the cost guards against, so the assertion is known to be able to fail. CI now runs the Postgres half of the encryption suite in its own job, a matrix over the supported PostgreSQL versions (17 on a pull request, 14 to 17 on a push, like the SQLx jobs), where it was one step of the single-version `rust-crates` job. Passed locally on 14, 15, 16 and 17. The page. `docs/reference/json-tree-leakage.md` is the one page a customer reads before putting a column on the type: what a stored row shows at rest, field by field; what each query type hands the database, tokens above all; what is authenticated; the side-by-side with v3; and the two options that change the picture. Refs #1324 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
🦋 Changeset detectedLatest commit: 17ac84f The changes in this PR will be included in the next version bump. This PR includes changesets to release 12 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
…to feat/json-tree-gates
407a9f5 to
e7f782d
Compare
fdf1eb9 to
9657508
Compare
Brings up the review fixes to the json_tree design record and its SQL prototypes from docs/json-index-stack-encrypt-spike. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Brings up the #1288 review fixes as #1329's branch carries them: the 127-level depth bound, the keyed member rank, the walk and rebuild bounds, the JsonTerms Debug, AtPath<Eq> as equality, the entry-versus-jsonb blockers and the changelog. No conflicts. With the rank keyed, the leakage reference's statement that nothing it describes reveals a key name holds for a token's holder too: the member order a token exposes is an order under the index key. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Brings up the array case of the rebuild-bound test from #1288, which the mutants gate asked for. No conflicts. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Carries the #1325 review fixes up through #1329. One conflict: both branches appended a PostgreSQL test at the end of the json_tree suite (this branch's planner test, the hostile-document test from below), and both are kept. Since this branch runs the suite in CI on every supported PostgreSQL, the hostile-document test now runs there too. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Brings in the #1329 review fixes (the selector-bound chain step and position gates, the refusal of positions in ordered paths, the JSON wire fuzz target), the key-built subtree wrapper, and the json plaintext held under Protected. One conflict, in packages/eql/tests/encryption/tests/json_tree.rs: both sides added a PostgreSQL test after postgres_extracts_walks_and_orders_by_path (this branch's postgres_the_planner_picks_the_indexes_without_help with its flat_document and nested_document helpers, and #1329's postgres_a_position_reaches_array_elements_only). Both are kept, the planner test first, each with its own body; git had interleaved their shared opening lines. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Brings in the JsonPath::covers test the #1329 mutants gate asked for. No conflicts. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Carries origin/next up from #1250 through #1329, so this PR reads against one merge base. Content-empty. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
coderdan
left a comment
There was a problem hiding this comment.
Review of what #1336 adds over #1329 (git diff origin/feat/json-tree-ordinal-chain...origin/feat/json-tree-gates), at d3a47b1. I checked every claim on the leakage page against stack-encrypt/src/json/{derive,mod,query}.rs and the v4 SQL (types.sql, functions.sql, operators.sql), confirmed the stored entry order on a live table, and ran the planner test on PostgreSQL 14, 15, 16 and 17.
The planner test and the CI job do what they say, with one gap in each. The leakage page is the problem. It is the page a customer's security reviewer reads, and several of its "not revealed" claims are false for the format as built. The common cause: sv (and q) are stored in depth-first pre-order, an object's members in sorted key order, under selectors that are the same in every row. Neither the page nor section 6 of the design takes that order into account.
High
- "Not revealed at rest" is wrong on four counts. The stored order gives away each row's exact shape, every node's parent (inferable across rows, and exact for top-level fields under the sealed root), every array's length (the count of its element selector), and the alphabetical order of sibling key names. The last is the open sorted-key-order leak from the #1329 review.
- The v3 comparison table claims v4 hides shape, parent relations and subtree sizes. It does not hide any of them: v4 shows shape equality just as v3 does, the parent relation is inferable in both, and a subtree's node count can be read off the order.
- The subtree-read bullet says the query sends "a token for the node." It sends the token of the top-level field the path starts with (or of the array element reached through positions right after it), so reading
$.profile.address.cityhands over$.profile's whole subtree in every row. A#ngate carries the position in the clear. Withsealed_rootoff, every such read sends the root token, which covers the whole column's shape. - "Tampering … never [changes] what a value decrypts to" is not true of extraction.
JsonNodeandJsonSubtreeopen each entry under thesandjthe server returns beside it, so->on$.salarycan be answered with the same row's$.bonus, or with the sealed root. Each value is genuine, but nothing checks it is the one at the path asked for. The fix is the owner's open decision. The page now says what holds today.
Medium
prow. "Direct members of the root carry nop" is true only with the sealed root on. With it on, it also delimits each top-level field's subtree, so that subtree's node count is readable exactly from one row.- Ordering. A bound's OPE term shows its exact place among the stored values, and whether it equals one, not "roughly where" it sits. A shared prefix of the encoding, not nearness in order, is what makes terms share leading bytes.
- CI:
rust-encryption-postgresis missing fromci-required'sneeds. The step it replaces sat insiderust-crates, which is listed. As things stand, a failing oracle or plan-shape test leaves the aggregate check green.
Low
- The lengths row leaves out the key-name length that container entries show, and the page never mentions
h(keyset id, data-key id, sealed-root flag). - The
validatejob's comment now sits above the new job. - The planner test's "TOASTed" pin is a size threshold (
pg_column_size > 2000), not a check of where the value is stored. The test also never covers a prepared statement past its fifth execution, which is whenplan_cache_mode = automay switch to a generic plan. That is the path a pooled driver takes. - The PR body summarises the page's at-rest claims as they were.
Checked and fine. COST 1000 on json_tree_query costs nothing at run time: the bitmap recheck calls it once per candidate row, and a projection or document-to-document @> only sees inflated estimates. The planner test depends on nothing that varies. ANALYZE reads every row, and the stats are over deterministic terms. Flat documents are about 1,150 bytes and every nested one is TOASTed (verified by chunk count). Under plan_cache_mode = auto the cache keeps custom plans for the nested table and uses a GIN-backed generic plan for the flat one. The control is meaningful. The matrix expression, service health check, EQL_TEST_DATABASE_URL form and the save-if: false restore of the rust-crates key are right. The removed rust-crates step is not duplicated. The changeset, the README link and the v3 "ordering terms" row are accurate (cipherstash-client 0.42.3 emits op on every scalar leaf, OPE or ORE by mode, with no per-path choice).
Left for the owner. Two things here are format decisions, so I have not changed the format for either. The first is the stored entry order: pre-order, with sorted keys. The second is that extracted entries open under the server's selector. Both are now stated on the page and recorded as open in the Phase 4 notes of the design document.
Fixes follow on this branch; each inline thread gets a reply with its commit.
| | `p`, the parent tag below the chain's heads | Nothing on its own. It is salted by the document's `iv`, so the same position in two rows is unrelated bytes. It shows **how many nodes are direct members of the root** (they carry no `p`); not which node is any other node's parent, not an array's length, and not whether two rows have the same shape. | | ||
| | `op`, the ordering term, only on leaves at the paths the index orders | **The order of that leaf's value among every row that holds one.** The term is order-preserving encryption under one key per column, so leaves at two ordered paths compare with each other too. An order-preserving term is not a random value: values that sort near each other get terms that share leading bytes. By default no path is ordered; see the options below. | | ||
|
|
||
| Not revealed at rest: any value, any key name, which node is whose parent, how long an array is, and whether two rows have the same shape. |
There was a problem hiding this comment.
[High] This sentence is wrong on four counts, not one.
sv stores the nodes in the walk's order: depth-first pre-order, the root first, each node followed by everything beneath it, and an object's members in sorted key order (for (key, child) in map over serde_json's BTreeMap). The selectors are the same in every row. So an observer with no key reads this off the stored rows:
- the shape of every row. The pre-order sequence of selectors fixes the tree exactly, so two rows have the same shape, key names included, if and only if their sequences are equal. On the plan test's tables every row has one sequence, as expected.
- which node is whose parent. It is not stored, but the parent path of each selector is fixed for the column, so comparing rows reveals it. Under the sealed root it is exact for the top-level fields: their entries are the ones with no
p, and each one's subtree runs to the next. - how long an array is. It is the number of entries that carry the element selector.
- the alphabetical order of sibling key names. This is the open leak from the feat(eql, stack-encrypt): chain json_tree tokens by ordinal and probe the walk to the first miss #1329 review, and it partly undoes what the keyed rank hides.
What this means for a user: a security reviewer who reads this sentence will sign off on a column, believing its rows' structure is private. It is not. The fix rewrites the sentence and the rows above it to say what the order reveals. Changing the order is a format decision for the owner, recorded in the design doc's Phase 4 notes.
There was a problem hiding this comment.
Fixed in 1f366a7. The s row now gives the count per selector and array lengths. A new row covers the entry order: shape, inferable parents and subtree counts, and the sorted order of sibling keys. The closing sentence lists what is and is not revealed at rest. The format is unchanged: the stored order is recorded as an open decision for the owner in the design doc's Phase 4 notes.
| | node count | yes, two entries per node | yes, one entry per node | | ||
| | size of every subtree | yes: each path entry's ciphertext is the subtree under it | no: only each scalar's length | | ||
| | parent and child relation | inferable from subtree sizes and co-occurrence across rows | hidden at rest; linkable only with a token the query supplies | | ||
| | whether two rows have the same shape | yes, selectors are column-deterministic | no, until a token is supplied | |
There was a problem hiding this comment.
[High] The v4 column of this table is wrong in three rows: shape (here), parent and child (line 48), and subtree size (line 47).
Selectors are column-deterministic in v4 exactly as in v3, and sv is stored in depth-first order, so shape equality is visible. The salt on p does not change that, because the order and the selectors give it away. Parent relations are inferable from the order across rows, and exact for top-level fields under the sealed root. A subtree's node count follows from the order, and its byte size is the sum of its entries' lengths. Section 6 of the design made the same claim before the entry order was considered.
What this means for a user: anyone comparing v3 to v4 concludes that v4 hides row structure. It does not. Fixed in the table, with an array-length row added, and noted under section 6 of the design doc.
There was a problem hiding this comment.
Fixed in 1f366a7. Shape is now "yes" for v4. Parent and child is "not stored, but inferable … exact for top-level fields under the sealed root". Subtree size is "no ciphertext spans a subtree, but its node count follows from the entry order". There is a new array-lengths row, and the subtree-read row names the top-level field's token. Section 6 of the design doc has a dated note on the correction.
| - **Containment (`@>`, `<@`).** The needle's own projection: the selectors of its paths, the value selectors of its leaves, and its element grouping. So the database learns which paths and which (path, value) pairs you asked about, and which rows hold them. | ||
| - **Path extraction (`->`, `->>`, `json_tree_path_query`).** The selector of the path asked. The entry returned is a ciphertext. | ||
| - **Ordering at a path (`<`, `<=`, `>`, `>=`, `ORDER BY ord_term(…)`).** The bound's own ordering term, so roughly where the bound sits in the order of the stored values at that path. | ||
| - **A subtree read (`json_tree_subtree`, `json_tree_subtree_at`, `json_tree_children`, `json_tree_child_count`).** A **token** for the node, and one gate (a selector or an array position) per step below it. A token is a capability over the structure of that subtree in *every row of the column*: whoever holds it can count the nodes under that path in any row, see how they nest and in what order, and derive the tokens of the nodes beneath. It still opens no value. Tokens are column-wide and do not expire, so a token sent once is useful to the database for as long as the column lives. Send a token to the database in a query and nowhere else. With the sealed root on (the default) the broadest token a query can send covers one top-level field's subtree; no query can walk a whole column. |
There was a problem hiding this comment.
[High] A subtree read does not send "a token for the node".
subtree_key_for derives the token of the deepest node the client can derive: the chain head (with the sealed root on, the top-level field the path starts with), then array positions immediately after it. Every key step below that is a gate. So json_tree_subtree_at on $.profile.address.city hands the server $.profile's token, a capability over the whole $.profile subtree in every row, forever. A #n:<selector> gate also carries the array position n in the clear. With sealed_root off the head is the root, so every subtree read whose path starts with a key sends the root token, the whole column's shape. The options section doesn't say this either.
What this means for a user: a team that reads a narrow path deep in a document, believing it reveals only that path, is in fact handing over the shape of the whole top-level field, or with the sealed root off the whole document. Fixed in this bullet and in the sealed_root option.
There was a problem hiding this comment.
Fixed in 1f366a7. The bullet says the token is the deepest node the client can derive: the top-level field, or the element reached through positions after it. It gives the $.profile.address.city example, notes that a #n gate's position is in the clear, and covers the root token with the sealed root off. The sealed_root option says the same.
|
|
||
| ## What is authenticated | ||
|
|
||
| Only the ciphertext `c`. Opening an entry verifies that it was sealed under this keyset, for this column, under this data key, at this path (`s`) and at this position in the document (`j`); a moved, swapped or re-labelled entry fails to open. The selectors, the projection, the parent tags, the ordering terms, and which entries a row holds are not authenticated, by the same rule every EQL stored value follows (ADR 0010 in `packages/stack-encrypt/docs/adr`). The consequence to design around: tampering with those fields changes **which rows a query returns or where a walk stops**, never what a value decrypts to. A query's result set is not integrity-protected; a decrypted value is. |
There was a problem hiding this comment.
[High] "never what a value decrypts to" does not hold for extraction.
JsonNode opens its entry under the s and j that arrive with it from the server (open_entry(&aes, &self.header, &aad, &self.entry, self.j)). Nothing compares s with the selector the client asked for, and the eql-bindings JsonTreeEntry path is the same. So a server can answer doc -> '<$.salary selector>' with the same row's $.bonus entry and its own s/j, and the client decrypts $.bonus's value as the salary. Returning the sealed root (j = 0) makes it decrypt the whole document. A walk's rows behave the same way. Each value is genuine and from that row, but which entry it is, is not checked. The client checks positions by place in sv only when it opens the whole document.
What this means for a user: a reviewer relying on "a decrypted value is integrity-protected" assumes -> returns the value at the path. A malicious or buggy server can swap fields within a row. Whether to check the returned selector is the owner's open decision (the node opens under the server's selector). The fix states the property as it holds today and records the decision in the design doc.
There was a problem hiding this comment.
Fixed in 1f366a7, as a statement of how things are today. The section says the client opens an extracted entry under the s and j returned with it, so a row's fields can be swapped ($.salary answered with $.bonus). Each decrypted value is genuine and from the row the result says. Checking the returned selector is the owner's open decision, recorded in the Phase 4 notes. No code changed.
There was a problem hiding this comment.
Follow-up in 17ac84f. My reply above says each decrypted value is "from the row the result says", and that is not right either. An extracted entry carries its own document's header, so the database can also list another row's entry from the same column in a row's place, and it opens. The page now says the value is genuine and from this column, and that both its path and its row are the database's word.
| | `s`, the path selector on every node | **Which paths the row has.** A selector is a keyed hash of the path, the same in every row of the column, so two rows with a selector in common hold the same path. Key names are inside the hash and not readable. Array elements share their array's selector, so `$.tags[0]` and `$.tags[1]` look alike. | | ||
| | `q`, the containment projection: one `{s, vs}` pair per node, nested by array element | **Whether two rows hold the same value at the same path**, through the value selector `vs`, a keyed hash of the path and the value. Not the value itself, and not whether two different paths hold the same value. **Which nodes sit inside the same array element**, through the nesting. That grouping is what makes containment exact for arrays of objects, and it is the one thing this format reveals that v3 did not. | | ||
| | the number of entries, and each ciphertext's length | **The node count**, and for each scalar the length of its value plus its key name, to within the AEAD's fixed overhead. A row's total size is the usual size signal any encrypted column gives. | | ||
| | `p`, the parent tag below the chain's heads | Nothing on its own. It is salted by the document's `iv`, so the same position in two rows is unrelated bytes. It shows **how many nodes are direct members of the root** (they carry no `p`); not which node is any other node's parent, not an array's length, and not whether two rows have the same shape. | |
There was a problem hiding this comment.
[Medium] The p row describes the sealed-root layout only, and understates what it shows there.
The root's direct members carry no p only when sealed_root is on (the chain heads are depth 1). With it off, the root has a token, its members carry a p, and only the root lacks one. And with it on, the p-less entries mark where each top-level field's subtree starts in the pre-order sv, so the node count of each top-level field's subtree is readable exactly from a single row. "Not an array's length, not whether two rows have the same shape" is true of p on its own, but in a table about what a row shows it reads as "not revealed", which F1 shows is wrong.
What this means for a user: someone sizing up what the default options disclose misses that they put every top-level field's size, in nodes, in plain view. Fixed in the row and in the sealed_root option's costs.
There was a problem hiding this comment.
Fixed in 1f366a7. The row now says which entries lack p under each option, and that with the sealed root on they delimit each top-level field's subtree, so its node count is exact. The sealed-root option lists this among its costs.
|
|
||
| - **Containment (`@>`, `<@`).** The needle's own projection: the selectors of its paths, the value selectors of its leaves, and its element grouping. So the database learns which paths and which (path, value) pairs you asked about, and which rows hold them. | ||
| - **Path extraction (`->`, `->>`, `json_tree_path_query`).** The selector of the path asked. The entry returned is a ciphertext. | ||
| - **Ordering at a path (`<`, `<=`, `>`, `>=`, `ORDER BY ord_term(…)`).** The bound's own ordering term, so roughly where the bound sits in the order of the stored values at that path. |
There was a problem hiding this comment.
[Medium] An ordering query reveals more than "roughly where the bound sits".
The bound's term is CLLW OPE under the column's key, deterministic and directly comparable with every stored op. The server learns the bound's exact place among the stored values at every ordered path (terms are keyed by column, not path), and whether it equals one of them. The op row on line 23 has a matching inaccuracy: terms share leading bytes when the values' encodings share a prefix (for a string, its first characters), not when the values "sort near each other". "az" and "b" sort adjacent and share nothing. ORDER BY ord_term(…) sends no term at all, so it doesn't belong in this bullet's list of what a query sends.
What this means for a user: someone weighing whether to order a path underestimates what a range query on it gives away. Fixed in both places.
There was a problem hiding this comment.
Fixed in 1f366a7. Ordering now says the bound's exact place among the stored values at every ordered path, and whether it equals one. The op row explains the leading bytes as a shared prefix of the encoding and adds equality. ORDER BY ord_term(…) is now noted as sending no term.
| |---|---| | ||
| | `s`, the path selector on every node | **Which paths the row has.** A selector is a keyed hash of the path, the same in every row of the column, so two rows with a selector in common hold the same path. Key names are inside the hash and not readable. Array elements share their array's selector, so `$.tags[0]` and `$.tags[1]` look alike. | | ||
| | `q`, the containment projection: one `{s, vs}` pair per node, nested by array element | **Whether two rows hold the same value at the same path**, through the value selector `vs`, a keyed hash of the path and the value. Not the value itself, and not whether two different paths hold the same value. **Which nodes sit inside the same array element**, through the nesting. That grouping is what makes containment exact for arrays of objects, and it is the one thing this format reveals that v3 did not. | | ||
| | the number of entries, and each ciphertext's length | **The node count**, and for each scalar the length of its value plus its key name, to within the AEAD's fixed overhead. A row's total size is the usual size signal any encrypted column gives. | |
There was a problem hiding this comment.
[Low] The length row covers scalars only, and the page never mentions h.
A container's ciphertext holds its kind, child count and label, so an object member that is itself an object or array shows its key name's length too. Lengths are exact once the fixed overhead (nonce, tag, framing) is taken off. Under the sealed root, the root's length is the document's length as JSON. The header h, stored in the clear, gives the keyset id, the data key's id and the sealed-root flag. The first paragraph's "nothing links a column to another column or another keyset" should be scoped to the terms, since the header names the keyset.
What this means for a user: small, but the page is meant to be complete field by field. Fixed by adding an h row and the container lengths.
There was a problem hiding this comment.
Fixed in 1f366a7. The lengths row includes a member's key-name length for every node (containers too), says lengths are exact less the fixed overhead, and notes the sealed root's document length. An h row covers the keyset id, the data key id and the sealed-root flag. The opening paragraph now scopes the no-link claim to the terms.
| # is a per-version property of the planner, which is why this is a matrix | ||
| # and not a step of `rust-crates`. PR and manual dispatch run PG 17 alone; | ||
| # a push runs the full set, like the SQLx jobs (see `setup`). | ||
| rust-encryption-postgres: |
There was a problem hiding this comment.
[Medium] This job is not in ci-required's needs: (line 1055), so its failure doesn't fail the aggregate check.
The step it replaces sat inside rust-crates, which is listed. After the move, a failing oracle test or plan-shape assertion turns this job red and leaves ci-required green. ci-required is the check meant to gate a merge, and it only reads the results of the jobs it needs. Nothing in pnpm run test:scripts catches this.
What this means for a user: once a branch-protection rule points at ci-required, a planner regression on PG 14 can merge with every required check green. Fixed by adding the job to needs, plus a guard in eql-suite-ci.test.mjs that every job in the workflow is listed (I checked it fails without the entry).
There was a problem hiding this comment.
Fixed in d01d547. rust-encryption-postgres is in ci-required's needs. eql-suite-ci.test.mjs gains "ci-required aggregates every job in the EQL workflow", which fails with rust-encryption-postgres named when the entry is removed. pnpm run test:scripts: 78 files, 1,371 tests pass.
| @@ -452,6 +452,68 @@ jobs: | |||
| # docs:validate + Clean-DB v3 install smoke. Both are version-relevant, so | |||
There was a problem hiding this comment.
[Low] This comment describes validate (docs:validate and the clean-DB v3 install smoke), but the new job was inserted under it. It now reads as part of the rust-encryption-postgres comment. Moved back above validate. The push paths comment ("The rust-crates job's encryption steps compile these by PATH") also now applies to this job, and says so.
There was a problem hiding this comment.
Fixed in d01d547. The comment is back above validate. The push paths comment names both jobs.
| .query_one( | ||
| &format!( | ||
| "SELECT avg(pg_column_size(doc))::int, | ||
| (SELECT count(*) FROM {table} WHERE pg_column_size(doc) > 2000) |
There was a problem hiding this comment.
[Low] Two gaps in what the planner test pins.
- "Every row TOASTed" is inferred from
pg_column_size(doc) > 2000, which measures a value's stored (compressed) size, not where it is stored. It is a sound proxy at today's sizes (flat about 1,150 bytes, nested about 28,600), but the assertion message claims more than it checks. Counting distinctchunk_ids in the table's TOAST relation is exact on PG 14 to 17 and costs one query. - Every query is a one-shot statement, planned with its operand in view. A pooled driver prepares once and executes many times, and after five custom plans
plan_cache_mode = automay plan generically, with the operand as an unseen parameter. That is exactly where the costsCOST 1000is meant to steer matter most. I measured it (PREPARE, six EXECUTEs, EXPLAIN EXECUTE). On PG 14 to 17 the cache keeps custom plans for the nested table and uses a GIN-backed generic plan for the flat one, so the indexes hold, but nothing pins it.
What this means for a user: gate 2 is about the plans customers actually get, and an ORM's prepared statement is the commonest way they get them. Fixed: exact chunk-count pin, and both queries are also asserted from a prepared statement past its fifth execution.
There was a problem hiding this comment.
Fixed in 491b09c. The pin counts distinct chunk_ids in the TOAST relation: 0 for flat, every row for nested. Both queries are also asserted from a SQL PREPARE executed six times, read back with EXPLAIN EXECUTE, with the selector kept literal as the functional index requires. Green on PostgreSQL 14, 15, 16 (docker postgres:N) and 17.10.
The leakage page said a stored row hides its shape, its parent relations, its array lengths and its key names, and that tampering never changes what a value decrypts to. Read against the code, none of that holds. `sv` and `q` are stored in depth-first pre-order, an object's members in sorted key order, under column-deterministic selectors. The order therefore fixes each row's shape, makes the parent relation inferable (exact for top-level fields under the sealed root), counts every array, and shows the alphabetical order of sibling keys. A subtree read sends the token of the top-level field the path starts with, not of the node asked for. With the sealed root off it sends the root's token. A position gate carries its position in the clear. An extracted entry opens under the selector the server returns with it, so a row's fields can be swapped. A bound's OPE term places it exactly among the stored values, not roughly. The page now states the format as built, field by field, including the header. The v3 comparison and the sealed-root option are corrected. The design document notes where section 6 was wrong, and records the entry order and the extraction selector as open decisions for the owner. The format is unchanged. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Moving the PostgreSQL half of the Rust encryption suite out of `rust-crates` into `rust-encryption-postgres` also took it out of `ci-required`. That job reads only the results of the jobs it needs, so a failing oracle or plan-shape test would have left the aggregate check green. The job is now listed. eql-suite-ci.test.mjs fails when any job in the workflow is missing from `ci-required`'s needs, which the old list would have tripped. The `validate` job's comment, which the new job had been inserted under, moves back above `validate`. The paths comment now names both jobs that compile the stack-* crates by path. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The planner test inferred "every row TOASTed" from pg_column_size, which measures a value's stored size, not where it is stored. It now counts the distinct chunk ids in the table's TOAST relation. That works on PostgreSQL 14 to 17, where pg_column_toast_chunk_id is 17 only. It also asked only one-shot statements, which are planned with their operand in view. A pooled driver prepares once and executes many times, and after five custom plans `plan_cache_mode = auto` may plan generically, with the operand as a parameter. Containment and the range at an ordered path are now also asserted from a prepared statement past its fifth execution, with EXPLAIN EXECUTE showing the plan the cache would run. Verified on PostgreSQL 14, 15, 16 and 17. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…isted under An extracted entry carries the header of the document it was sealed in, and the client opens it under that header. A database can therefore list another row's entry, from the same column, in a row's place, and it opens. The page said a decrypted value "comes from the row the result says". It now says the value is genuine and from this column, and that both its path and its row are the database's word. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 17ac84fa6a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - **Planner fragility (gate 2).** `eql_v4.json_tree_query` is plpgsql with `COST 1000` so that a bitmap scan over the GIN index beats a sequential scan even when the documents are TOASTed and the heap looks like a few pages. That was observed in the benches, at 10,000 and 100,000 rows, on one PostgreSQL version, and nothing held it. `postgres_the_planner_picks_the_indexes_without_help` in the encryption suite now does: on a flat table (1,000 four-node documents, every row inline) and a nested table (300 documents of about 220 nodes, every row TOASTed, both regimes pinned by `pg_column_size`), with every planner setting at its default and after `ANALYZE`, containment plans as a bitmap scan over the GIN index and a range at an ordered path as a scan of the functional btree on `ord_term(doc -> selector)`. A control sets the function's cost to 1 for one transaction and shows the planner taking the sequential scan on the nested table, so the assertion is known to be able to fail. The suite runs in `test-eql.yml` in a new `rust-encryption-postgres` job, a matrix over the supported PostgreSQL versions (17 on a pull request, 14 to 17 on a push, like the SQLx jobs); it was a step of the single-version `rust-crates` job. Measured locally on 14, 15, 16 and 17 before the commit. | ||
| - **Leakage (gate 3).** `packages/eql/docs/reference/json-tree-leakage.md` is the one page: what a stored row shows at rest, field by field; what each query type hands the database; what is authenticated; the side-by-side with v3 from section 6 above, brought up to date (no `g`, position-keyed `p`, ordering opt-in) and corrected where section 6 was wrong (the entry order; see the note under its table); and the two options that change the picture. It says what a token is and that it is column-wide and does not expire, which is the sentence a customer most needs before sending one. | ||
| - **Open for the owner: the stored entry order (2026-10-23, review of #1336).** `sv` and `q` hold the nodes in the walk's order, depth-first, an object's members in the sorted order of their keys (serde_json's `Map` without `preserve_order`; a build that unifies that feature on would store insertion order instead, which leaks the application's field order). Across rows this reveals each row's shape, its parent relation, every array's length and the alphabetical order of sibling key names, which partly undoes what the keyed rank (decision 11) hides from a token holder. The leakage page says so. Closing it is a format change, not a documentation one: a stored order that does not follow the tree (a document-keyed permutation, say, with `j` the permuted position), a rebuild that does not read the tree from the order, and the same for the pairs of `q`, whose nesting would still group array elements. Not taken here. | ||
| - **Open for the owner: an extracted entry opens under the selector the server returns with it.** `JsonNode` and `JsonSubtree` open each entry under the `s` and `j` that arrive beside it, so a decrypted value is always one sealed in that row but not necessarily the one at the path asked for (`->` on `$.salary` can be answered with the row's `$.bonus`, or with the sealed root, which opens to the whole document). The leakage page states this under "What is authenticated". Checking the returned `s` against the selector the client derived would close it for `->`; a walk's rows would need each child's selector checked against its parent's path. |
There was a problem hiding this comment.
Correct the row-level integrity guarantee
When an untrusted database substitutes an extracted entry from another row in the same column, that entry brings its source h, s, and j; entry_aad binds the column, selector, ordinal, and data-key tag, but no row identity, so the entry can decrypt successfully. Saying the value was always sealed in “that row” understates the cross-row substitution risk and contradicts the new leakage reference; only the column is guaranteed, while row provenance remains the database's word.
AGENTS.md reference: AGENTS.md:L684-L692
Useful? React with 👍 / 👎.
cipherstash-bot
left a comment
There was a problem hiding this comment.
Recommendation: 🟡 merge after changes
The planner test must also check a forced generic plan before merging (see the inline comment). The rest of the PR can merge as it is.
Other findings not posted as comments
- Optional: the six
EXECUTEruns of each prepared statement discard their rows. A generic plan that returns wrong rows would pass. Compare the row count with the one-shotcount(*).packages/eql/tests/encryption/tests/json_tree.rs:1135 - Optional: the control block covers only containment on the nested table. No control shows that the
_ordbtree assertion can fail.packages/eql/tests/encryption/tests/json_tree.rs:1158 - Optional: no script test checks that
test:encryption:postgresruns once for each version insetup'spg-versions. If the step moves back into a job with onepostgres:17service, the existing guards inscripts/__tests__/eql-suite-ci.test.mjsstill pass. Add a case that finds the job running the task and checks its matrix and service image..github/workflows/test-eql.yml:463
How this review was made
| Agent | Model | Review type | Result |
|---|---|---|---|
| claude | claude-opus-5-5 | test-gap | 2 found, 1 posted |
| codex | gpt-5.6-terra | test-gap | 1 found, 1 posted |
Synthesis: claude-sonnet-5-5 merged the findings, removed duplicates and dropped findings it could not confirm in the code. 1 posted finding(s) were raised by two or more models.
Plain language: claude-sonnet-5-5 read every comment as a new reader would. 1 comment(s) had a problem that stopped the reader acting; it rewrote 1.
Stack: 5 earlier and 0 later pull request(s) (#1229, #1250, #1288, #1325, #1329, *️⃣ #1336). *️⃣ marks this pull request.
Context loaded: the description, 1 linked issue(s) and 25 discussion entries.
| ("contains", &containment, "_gin", "containment"), | ||
| ("range", &range, "_ord", "a range at an ordered path"), | ||
| ] { | ||
| for _ in 0..6 { |
There was a problem hiding this comment.
Change before merge: the prepared-statement check does not force a generic plan, so it may never test the generic plan for the nested table.
Impact: With plan_cache_mode = auto, PostgreSQL keeps a custom plan when it costs less than the generic plan. For the nested table, the cache may keep custom plans. Then EXPLAIN EXECUTE shows a custom plan, the same as the one-shot query. If the generic plan for that table becomes a sequential scan, no test fails. A pooled driver can reach that generic plan.
Evidence: The test sets no plan_cache_mode, so it uses auto. It runs six EXECUTE calls and then reads EXPLAIN EXECUTE, which shows the plan the cache would run next. The file v4_range.rs:566 already uses SET plan_cache_mode = force_generic_plan for this case.
Fix: After the six EXECUTE runs and the auto assertion, run SET plan_cache_mode = force_generic_plan. Run the same EXPLAIN EXECUTE and assert the same index name. Then run RESET plan_cache_mode. Do this for both the contains and range statements on both tables.
Found by 2 models: claude, codex
Summary
EQL (Encrypt Query Language) is the SQL bundle that stores and queries encrypted values in PostgreSQL; EQL v4 is its version for values written by Stack Encrypt, the Rust encryption engine in
packages/stack-encrypt. Itsjson_treetype (unreleased) stores an encrypted JSON document as one entry per JSON node and answers containment, path extraction, ordering and subtree reads without the key. Three items gate its release: the stored document's size, the query planner's reliability, and a leakage write-up. The size work landed in #1329. This PR closes the other two.The planner. Containment over a
json_treecolumn runs througheql_v4.json_tree_query, a plpgsql function that returns the document's stored projection and is priced atCOST 1000so PostgreSQL prefers the GIN index over scanning every row. Large documents are stored out of line (TOAST), which makes the table's main heap look tiny and a sequential scan look cheap; the cost is what keeps the index in play. The benches had shown the right plan on one PostgreSQL version and nothing held it. A new test now does, with every planner setting at its default, on both storage shapes, and CI runs it on every supported PostgreSQL version.The page.
packages/eql/docs/reference/json-tree-leakage.mdsays what a stored row shows the database at rest. That covers: which paths, how many nodes at each, and so every array's length; which (path, value) pairs; which nodes share an array element; lengths; the order of values, only at paths the index orders; and the row's shape. The shape comes from the stored depth-first entry order, which also shows parent relations and the alphabetical order of sibling keys. The page also covers what each query hands the database: a subtree read hands it the token of the top-level field the path starts with, which is column-wide and does not expire. It says what is authenticated (only the ciphertext; an extracted entry opens under the selector the server returns with it). And it compares all this with EQL v3'sste_vec. Two of these, the entry order and the extraction selector, are open decisions for the owner, recorded in the design document.Changes
packages/eql/tests/encryption/tests/json_tree.rs):postgres_the_planner_picks_the_indexes_without_help. Flat table of 1,000 four-node documents (every row inline) and nested table of 300 documents of about 220 nodes (every row TOASTed), both regimes pinned; afterANALYZE, containment plans asBitmap Index Scanon the GIN index and a range at an ordered path as a scan of the functional btree oneql_v4.ord_term(doc -> selector). Both are asserted as one-shot statements and from a prepared statement past its fifth execution. The TOASTed regime is pinned by counting distinct chunk ids in the TOAST relation. A control sets the function's cost to 1 inside a transaction and asserts the sequential scan, so the test is known to be sensitive to the cost it guards..github/workflows/test-eql.yml): newrust-encryption-postgresjob, a matrix oversetup's PostgreSQL versions (17 on a pull request or manual dispatch, 14 to 17 on a push), runningmise run test:encryption:postgresagainst apostgres:<version>service; the step and the service are removed fromrust-crates, which keeps the DB-freetest:encryptionand the WASI check. Restores therust-cratescache key withsave-if: false. Listed inci-required'sneeds, whicheql-suite-ci.test.mjsnow holds for every job in the workflow.packages/eql/docs/reference/json-tree-leakage.md, linked frompackages/eql/docs/README.md; a Phase 4 note in the design document.@cipherstash/eqlpatch (a docs page in the bundle).Verification
mise run test:encryption:postgres(installs the v4 bundle built from this tree) passed against PostgreSQL 17.10 (Homebrew) and against thepostgres:14,postgres:15andpostgres:16images frompackages/eql/tests/docker-compose.yml; the new test runs in about 9 to 11 seconds, sealing included.cargo clippy -p eql-encryption-tests --testsandcargo fmt --checkclean;mise run test:doc-anchorspassed (117 links).pnpm run test:scripts: the workflow guards (eql-suite-ci,eql-workflow-filters,workflow-working-directory,workflow-mise-cache-image) pass with the new job.tests-crates.ymlandtest-eql.ymldispatched by hand on this branch, since GitHub has not been firingpull_requestruns for this repository since 2026-10-22; results will be noted in a comment.Review fixes (1f366a7, d01d547, 491b09c):
pnpm run test:scripts(78 files, 1,371 tests), the fullmise run test:encryption:postgreson PostgreSQL 17.10, the json_tree suite onpostgres:14,:15and:16, and the EQL static checks (docs:validate:*,test:lint,test:schema,test:doc-anchorsand the rest) all pass.Related
feat/json-tree-ordinal-chain.Review notes
ste_vecbetter than the document does.https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu