Repository navigation
Conversation
…n-list merge The test module #1366 added to `tests/json_index.rs` was written against the gated walk: it called `subtree_nodes(doc, token)`, which the token-list read (#1363) replaced with `read_rows(doc, key)`, and opened an extracted element with `decrypt_as::<Value, JsonNode>`, which has no implementation (an extracted entry opens through `JsonExtracted`, which checks its selector first). Merged on top of #1363 the file no longer compiled, and the `tests-crates` workflow is red on `next` at ebdb5de. Both calls now use the helpers the rest of the file uses. The test's assertions are unchanged. Refs #1366, #1363. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
… into its AAD A spike, not a change to land as it is. It evaluates the proposal that a client should be able to verify a returned subtree of an encrypted JSON document against a database that may rewrite, drop or rearrange rows, without reading the whole document. Every node of a json_tree document is its own AES-GCM ciphertext. Today its AAD binds the keyset, the column, the data key, the node's selector and its position in the stored list, and the client checks a subtree read by requiring its rows to be the consecutive positions below the node asked for. That check is sound only because the stored order is the depth-first order of the tree, which is the order #1357 wants to stop storing. With this change a node's AAD also binds a digest of its children's AEAD tags, in the order the parent tags enumerate them, so a container's ciphertext commits to exactly the children it was sealed over, and through their tags to everything beneath them. Sealing runs from the leaves up. Opening places the rows by the chain the server walks (the self tags and parent tags, which the client derives from the tokens it sent), then opens each node under the digest of the children the chain gave it: a child swapped, dropped, added or moved under another parent fails at the cipher, in whatever order the rows arrive. The consecutive- position rule is gone, and nothing reads the stored position as document order any more. What it does not do: a node's own placement in the document is still checked only by its selector and label, since verifying it against an ancestor needs that ancestor's entry, which the database cannot locate without the ancestor's token (and sending that token is the capability the token-list read removed). The sealed root commits to no children, because the budget fallback returns it alone. Both are discussed in the write-up this spike accompanies. Costs: nothing stored changes; the AAD is not stored. Opening a document with its root open now derives the root's token, so it needs the keyset loaded; a `->` extraction of a container with children no longer opens (it was refused by kind before, and is refused by the cipher now). The AAD domain is bumped to `json/entry/v2` so a v1 ciphertext fails loudly. Tests: the stack-encrypt JSON suites pass; the hostile-row tests still refuse every case, several now at the cipher rather than at the position check, and their assertions accept either. Refs #1357. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The second half of the spike (#1357): with the cascading tags placing a read's rows by the chain, nothing reads the stored position as document order, so the order can be random. Behind `STACK_ENCRYPT_JSON_SHUFFLE=1` (a process-wide switch, so every existing test and the PostgreSQL oracle run under it unchanged; a real change records it like `sealed_root`, or is simply the format), sealing draws a uniform permutation of the slots `1..n` from a CSPRNG, keeps the root at slot 0 (where the budget fallback and a whole-document open look for it; every document has a root, so its slot says nothing), binds each entry's AAD to its slot, and shuffles the pairs and element groups of the containment projection, which `jsonb @>` reads as sets. No SQL changes. The database never derives the permutation: it finds entries by tag and returns the slot, and containment is set containment. What the stored order revealed (the tree, every node's parent, subtree sizes, the byte order of sibling keys) it no longer does; the multiset of selectors, the element grouping of `q`, which entries are heads and the ciphertext lengths remain, as the report measures. Under the switch the PostgreSQL oracle passes 12 of 14 (the two compare slot numbers with depth-first positions) and the stack-encrypt suites 513 of 543 (slot-numbered tests, pinned vectors and the leakage calibration, which measures that the order reveals the shape). No hostile row opens. Refs #1357. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
|
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
Contributor
Mutation testing (cargo-mutants,
|
| caught | missed | unviable | timeout |
|---|---|---|---|
| 27 | 7 | 22 | 0 |
Surviving mutants: add a test that fails on each before merging
packages/stack-encrypt/src/json/derive.rs:948:5: replace shuffled -> bool with false
packages/stack-encrypt/src/json/derive.rs:950:90: replace == with != in shuffled
packages/stack-encrypt/src/json/derive.rs:1655:16: delete ! in Placing<'_>::node
packages/stack-encrypt/src/json/query.rs:51:9: replace JsonProjection::shuffle with ()
packages/stack-encrypt/src/json/query.rs:812:5: replace fisher_yates with ()
packages/stack-encrypt/src/json/query.rs:813:39: replace + with - in fisher_yates
packages/stack-encrypt/src/json/query.rs:813:39: replace + with * in fisher_yates
…pers Three jobs were red on the spike branch. Clippy under `-D warnings` named a redundant closure around `SafeRand::from_entropy` and `fisher_yates` sitting after the test module in `query.rs`; both are moved or rewritten as it asked. The Go job's drift check found the encrypt guest's `Cargo.lock` without the `vitaminc` dependency the spike added to `stack-encrypt`; the lock is regenerated (one line). The mutants gate missed a mutant that replaces `aead_tag` with a single zero byte, which sealing and opening both tolerate because they share the helper; two unit tests now pin the tag to the last sixteen bytes of a ciphertext and the children's digest to those tags, in order, and nothing else. Refs #1357. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Contributor
Author
|
Appended
Nothing else changed; the results in the description stand. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A spike, not a change to merge as it stands. It tries two ideas on the EQL v4
json_treeformat (EQL is the SQL library installed into a customer's PostgreSQL;json_treestores an encrypted JSON document as one small AES-GCM ciphertext per node, plus search terms) and reports what they cost and what they buy:qis shuffled, so the stored order no longer reveals each row's tree, the parent of every node, subtree sizes or the byte order of sibling key names. Behind a process-wide switch (STACK_ENCRYPT_JSON_SHUFFLE=1) so every existing test and the PostgreSQL oracle run under it unchanged.The two compose: the cascade is what lets the client place rows without reading the stored order, which is what the shuffle takes away. The write-up with the analysis, the measurements and the recommendation is in the report linked from #1357.
Changes
packages/stack-encrypt/src/json/derive.rs: sealing runs from the leaves up;entry_aadtakes a digest of the children's tags (AAD domain bumped tojson/entry/v2); opening a subtree read places rows by the chain's tags (self tags, parent tags, the member and element rules) and opens each node under its children's digest, in whatever order the rows arrive; the consecutive-position rule is gone. Opening a document whose root is open walks the chain the same way, so it needs the keyset loaded (StackCipher::loaded_keyset, new,pub(crate)). The random stored order isslots()plusJsonProjection::shuffle.packages/stack-encrypt/src/cipher.rs:loaded_keyset.packages/stack-encrypt/src/json/query.rs:JsonProjection::shuffleandfisher_yates.packages/stack-encrypt/Cargo.toml, both lockfiles:vitaminc(forSafeRand).tests/json_read_checks.rs,tests/json_index.rsand the Postgres oracle accept a refusal at the cipher as well as aJsonWrongEntry, since the cascade moves several refusals from the position check to the AEAD. Extracting a container with children now fails at the cipher (its AAD covers children->does not return); it was refused by kind after opening before.tests/json_index.rsonnextdoes not compile (the fix(stack-encrypt): open JSON index floats to the exact value sealed #1366 module calls a helper feat(stack-encrypt, eql): json_tree tokens derivable from the path, a self tag, client-side wildcards #1363 removed); fixed with the helpers the rest of the file uses.tests-cratesis red onnextat ebdb5de because of it.Verification
Run on this branch, locally:
cargo nextest run -p stack-encrypt --all-features(JSON suites and the lib): 531 passed, 0 failed, with the cascade and the default stored order.packages/eql/tests/encryption/tests/json_tree.rs, EQL v4 bundle built from this tree, PostgreSQL 17.10): 14 of 14 pass with the cascade: every read equalsjsonb #>/jsonb_path_query, containment equalsjsonb @>, and every tampered answer is refused.STACK_ENCRYPT_JSON_SHUFFLE=1: the oracle passes 12 of 14; the two failures compare returned slot numbers with depth-first positions (j = 2where 8 was expected), not answers. The stack-encrypt suites pass 513 of 543; the 30 failures are tests that pick an entry by its depth-first slot, the pinned derivation vectors (the order is random now), theq-equals-needle wire test, and the leakage-calibration tests that measure that the order reveals the shape, which it no longer does. No hostile row was accepted in any run; one apparent acceptance (a_row_reported_at_the_wrong_depth_is_refused) was the test's slot-numbered row landing on a node at that depth, checked with a throwaway test that shifts every row's depth and is refused every time.cargo fmt --checkclean;cargo clippy -p stack-encrypt --all-features --testsclean.Related
docs/plans/2026-10-17-json-tree-index.md; leakage page:packages/eql/docs/reference/json-tree-leakage.md.Review notes
entry_aad,sealandPlacing::nodeinderive.rs.json_tree_path_query_firstsemantics at a wildcard path under a random order.https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu