Repository navigation
test(stack-encrypt): check the json_tree leakage page against the scheme - #1362
Conversation
The leakage page for EQL v4's json_tree index states in prose what a database holding the column learns at rest and from each query. Nothing held the code to it: a leak the page leaves out would pass every test. This adds an executable form of the page. `profile.rs` writes each of its sentences as a clause of a leakage function over the plaintext alone, with every keyed value replaced by an opaque name that is equal exactly where the real bytes are. A simulator builds a wire-shaped column and query transcript from that function's output and seeded randomness only, chaining parent tags with the real hash below every token the profile says a query reaches. Twenty-seven distinguishers, each an exact feature an observer of the column could compute (including the database's own walk of the parent tags from a token it was sent), compare the simulated column with one the crate actually sealed under the fake key source. Three kinds of check make that mean something: the full profile is indistinguishable from the real scheme on handcrafted and 48 random columns; each of 34 mutant profiles, one clause dropped, is told apart by at least one distinguisher; and the page as written (without the one clause it lacks) is told apart by exactly the distinguisher that sees the missing clause. What it found: an ordering term's length (65 bytes for a number, 8 * (len + 1) + 1 for a string) is not on the page; the page's summary "not revealed at rest: whether two different paths hold the same value" contradicts its own op row, since equal values at two ordered paths have equal terms; and "sorted order of their key names" is UTF-8 byte order, not jsonb's. The two issues it was built to check (#1357, the stored order reveals shape; #1360, a subtree read's token reaches its whole top-level field) come out as the page states them. The page is left unchanged. The profile and the real-side adapter are expected to change with the 2026-10-24 design change (ordinal-free member tokens, a self tag, no gates); the module docs say how. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
Mutation testing (cargo-mutants,
|
…e-leakage-profile Brings in the review fixes on #1358: a walk cut short is JsonWrongEntry, a walk with no gates refuses an array that does not start at element 0, and the leakage page says what that walk still does not check. No conflicts; the leakage profile's tests pass unchanged. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…e-leakage-profile Brings in next after #1345, through #1358's merge of it. Conflict: packages/stack-encrypt/Cargo.toml only. Both sides added the same proptest dev-dependency line (#1345 for the generated containment property, this branch for tests/json_leakage) with different comments. Kept the one line and one comment naming both uses. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…akage-profile Brings in #1363's format (a self tag on every entry, ordinal-free member tokens, client-side wildcard expansion with a token list, no gates) so the leakage profile can be ported to it. The only conflict was the comment on stack-encrypt's proptest dev-dependency, which both branches use; one line, one comment naming both users. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…format
The profile was written against entries {s, c, p?, op?}, member tokens
carrying their ordinal, and a subtree read that sent one head token plus
plaintext gates. #1363 adds a self tag x to every entry, drops the
ordinal from member tokens, and has the client send the tokens of
exactly the nodes a read names, wildcards expanded to a bound with a
probe per expanded array. Run unchanged against it, the module did not
compile (a key now holds a list of tokens), and with that patched the
positive control failed on wire_fields, token_reach, token_derivability,
gated_walk_results and transcripts while every mutant still read as
caught, which is what a broken positive control looks like.
L now states the self tag (one fresh name per entry, so unlinkable
across rows and distinct within one), each read's token list with the
node each token names in each row and what it reaches below, the probe
flag, the budget, and the member or element rule each reached node's
token takes. The simulator programs x and p with the real hash only
where a sent token links them. Of the 27 distinguishers 26 are kept and
gated_walk_results becomes subtree_results (with the sealed-root budget
fallback); eight are new (stored_lengths, self_tag_equalities,
token_counts, bound_recovery, token_hits, wildcard_positions, hit_rules,
ancestor_links). The mutants go from 34 to 44: the two gate mutants are
gone, four are redefined for token lists, and twelve are new, among them
the pre-#1363 top-level token and member rule. All 44 are caught, the
positive control passes on the handcrafted columns and 48 random ones,
and string ordering terms are now modelled folded, as the code and the
page have them.
One finding against the page as #1363 published it: the database's
member-rule-then-element-rule test on every reached node tells it
whether each node below a token is an object member or an array
element, which nothing stored says. The page now says so, and
the_hit_rule_separates_it_from_the_real_scheme keeps the sentence
load-bearing. The SQL comment that the server cannot tell a probe from
any other token is not true of its place in the list or of what it
finds; a_read_gives_away_its_bound_and_its_probes shows it.
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
… node token_derivability derived member and element children of every token a read sends, which for a wildcard at the default bound is 65 tokens, most of them naming positions no row has, and their children name nothing either. Deriving only from tokens that find a node in some row halves the calibration's time (14.8 s to 9.6 s) with the same 44 of 44 mutants caught by the same distinguishers. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…1363 The leakage profile in #1362 was written against the format before #1363 and has been ported to it. This records, from the code at 9a7ca37 and at the port, every clause, distinguisher and mutant of the two profiles side by side; the failures the old test showed on first meeting the new format and what each one taught; the calibration before and after; and what the database learns now that it did not (the self tag, the token count and the bound it gives away, the probes, the member-or-element rule under a token) and what it no longer learns (an ancestor's capability in every row, the gates with their plaintext positions and selectors, the head token). Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…akage-profile Brings in b992e78, #1363's correction to the v3 column of the leakage page's comparison table. It touches a different part of the page from the sentence this branch adds, and no code. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
|
Read the tables as a measure of the profile, not of the scheme. More clauses, distinguishers and mutants mean more of what the database sees is written down and tested; they do not mean the format leaks more. Net, the format leaks less, and where it matters: a read no longer sends any token above the node it asks for (the old deep read sent an ancestor's token, a capability over that whole field in every row for the life of the column), and the gates, which spelled each step and position out in the clear, are gone. What is new is small: the self tag Ported this profile from the json_tree format it was written against to #1363's, and retargeted the PR onto #1363's branch. The full write-up, with the first run's failures and what the database learns now and no longer learns, is The clauses, before and afterThe "before" column is the profile at "Not measured" in the before column means the old profile had no such clause, for one of two reasons, and the row says which: the old format had no such thing to state (
The distinguishers, before and after27 before, 35 after: 26 kept unchanged in meaning, 1 renamed, 1 removed (by the rename), 8 added. The walk-based ones were rewritten for the new chain, but what each measures is the same.
The mutants, before and after34 before, 44 after: 28 kept unchanged, 4 redefined for token lists, 2 removed, 12 added.
Calibration, before and after
|
…lement Two sentences the leakage-profile port (#1362) showed to be wrong. The json_tree_subtree comment said the server cannot tell a probe from any other token: it can, since the probe is the last token of its level, the token count gives the bound away, and a hit on it carries the element's selector rather than the leaf's; the executable profile pins this. And the leakage page's q row still called element grouping the one thing v4 reveals that v3 did not, after the comparison table below it was corrected to say v3 reveals it through document order and the a flag. Comments and prose only; no SQL body changes. Refs #1362, #1363 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The note's tables grow from the old profile to the new one, and read as a worsening to a reader who stops there; the owner read them that way. The conclusion was in the last two sections. It now opens the note: the growth measures the test, and the format discloses less, with the ancestor capability and the gates gone and the additions small. Refs #1362 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…e is An old-profile clause marked (none) was either a thing the old format did not have, or a thing it had that the old profile failed to model; the table did not say which, and the owner asked. Each row now says, and a paragraph above the table names the method's limit: a clause the profile and the distinguishers both omit is invisible to the calibration. Refs #1362 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…(none)' '(none)' read as 'no leakage'. The cell says what is true of the old test: it did not measure the clause, with the reason beside it. Refs #1362 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…akage-profile Takes #1363's probe-exception sentence on the wildcard bullet and keeps this branch's hit-rule sentence on the token bullet. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Review at high effort: findingsReviewed What I checked: the real scheme ( The simulator does not cheat.
The hit-rule sentence the port added to the page is exact. It covers every node beneath a sent token, not the named node itself, which is found by I'll fix 1 to 10 on this branch, except the page wording in 3 and 7, which I'll leave for the owner. |
…leakage profile
A CLLW OPE term is a carry byte, then one byte per plaintext bit: the
PRF of the bits before it, plus the bit at weight 128, plus a carry of
zero or one from the right. Two terms therefore agree to within one at
every byte before the first bit their encodings differ in, and differ
by 127 to 129 there. The database reads the exact common-prefix length
of every pair of values off the stored terms; for numbers that is their
signs and binary exponents.
The page states the disclosure ("the term also shows how long that
common prefix is"), but the profile listed it as out of reach on the
ground that it was not an exact function of the plaintexts. That holds
for the byte prefix, which carries move, and not for the bit prefix. Its
simulator drew terms with no prefix structure at all, so a profile that
left it out passed.
The profile now states the bit LCP of every two classes adjacent in
rank (Leakage::ordering_prefixes, from vitaminc_ore's encodings). The
simulator builds CLLW-shaped terms under its own key over codes that
have exactly those prefixes, and ordering_term_prefixes reads them. The
mutant ordering_prefixes_hidden is caught by that distinguisher alone.
The page's sentence says bits, and what they mean for a number and a
string.
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Every feature named selectors, value selectors, tokens, keysets and ivs
in a space of its own, and self_tag_equalities compared x only with x
and p. A token equal to its path's selector, a value selector equal to a
selector, or a self tag equal to the front of a token would be visible to
the database, and no feature would change.
cross_kind_equalities reports every pair of kinds that share a value: the
sixteen-byte values with each other, and the eight-byte tags against the
first eight bytes of each. It is empty on the real scheme and in the
simulator. No mutant can state such a world, since the profile names each
kind on its own, so a harness test shows it fires: replacing one read's
token with a stored selector gives exactly ("s", "token").
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…nd projection wire_fields covered the keys of rows and entries only. The parser reads t, budget, sv, el, alt, s and op out of a query term and the projection, and drops anything else unread, so a term that also carried its bound or its path would have left every feature unchanged. term_fields reports every key path inside each row's q and each query term as sent. needle_alt_dropped, needle_flat and projection_flat are now caught by it as well as by their own features. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
… index 3 The database derives an element's token from its array's at any index. The feature tried indices 0 to 3 only, so a read whose tokens all sat at index 4 or more and found nothing (`$.tags[10]` beside `$.tags`, in a column whose arrays are shorter) read as unlinked. A profile that unchained such a token would have passed it. It now also tries every index up to the longest token list sent or 64, whichever is more, under the selector of the children each found token's walks reach: the one selector an element under it can carry, so the check stays cheap. The ordered-strings column reads `$.tags[10]`, and a harness test holds that the real scheme's tokens link it to `$.tags`. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…s a mutant each Four clauses had no mutant, so nothing showed the calibration could see them: the key tag's length, an ordering bound's selector, an element's ordinal being its index (element_tokens_without_index changes the names of tokens sent, not the stored chain), and each node's parent. key_tag_shortened is caught by headers and stored_lengths, order_selector_fresh by transcripts and ordering_matches, element_ordinals_reversed by ordinal_patterns among others, and parents_lifted by token_reach and parent_tag_equalities among others. Also drops mutants.rs's repeated "At rest" banner. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
… exercised Several clauses the positive control should hold were never tested. No ordered path held a null, a boolean or a container, so "op only on number and string leaves" was never checked. No ordered number was past 2^53, where 9007199254740992 and 9007199254740993 share a term under two value selectors. The random columns held only lower-case ASCII strings and never ordered strings, so folding and multi-byte lengths showed up in one handcrafted column alone. The random generator now holds big integers, floats, and strings that fold or are wider in bytes than in characters. It keeps non-scalars at the ordered paths, and orders numbers or strings by a coin toss (one kind per world, as the profile requires). It runs 256 cases instead of 48, in about three seconds. A small handcrafted column orders numbers of both signs and many magnitudes, the two big integers, and leaves that carry no term, which also gives the shared-prefix clause something to bite on. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…assertions messages After the merge of #1363, json_tree_subtree's comment says the server "can pick it out"; mod.rs and the doc comment of a_read_gives_away_its_bound_and_its_probes still quoted the old claim as standing. Both now say it was corrected. The test's assert_eq! calls gain the messages the rust-testing convention asks for, and mod.rs names the two harness tests that show features no mutant can reach do fire. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The net paragraph said the member-or-element rule "was already disclosed by the gated walk's ordinals and #n: gates". Before #1363 every child's token took one rule, and an only child's ordinal is 0 either way, so a node that is an only child in every row had its kind disclosed by nothing. The note's own finding said so. The paragraph now says what was and was not disclosed, and that the net rests on the ancestor token and the gates. The SQL-comment paragraph records that #1363 corrected the comment, and a closing section records what the review added: the ordering terms' shared prefixes, which both profiles missed, the new checks, mutants and corpus, and what is left to the owner. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…akage-profile Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Review fixes pushed: head
|
| finding | fate | commit |
|---|---|---|
| 1 [High] ordering terms' shared prefixes | fixed: clause Leakage::ordering_prefixes, CLLW-shaped simulated terms, distinguisher ordering_term_prefixes, mutant ordering_prefixes_hidden (caught by that distinguisher alone), page sentence now in bits |
750960062 |
| 2 [Medium] no cross-kind equalities | fixed: cross_kind_equalities, and a harness test that it fires on a token that is a stored selector |
076a9212e |
| 3 [Medium] numbers and strings at one ordered path | recorded under "Out of reach" in mod.rs with the measurement (750960062). Left to the owner: what the page should say, after which the profile can state it |
— |
| 4 [Low] term and projection field names | fixed: term_fields |
e52fe64f3 |
5 [Low] token_derivability below index 4 |
fixed: also the element rule up to max(64, longest list) under the found node's children's selector; the ordered-strings column reads $.tags[10], and a harness test holds the link |
6b3846320 |
| 6 [Low] four clauses without a mutant | fixed: key_tag_shortened, order_selector_fresh, element_ordinals_reversed, parents_lifted, all caught |
7ca5b0df2 |
| 7 [Low] corpus gaps | fixed: big integers, floats, strings that fold, non-scalars at ordered paths, random columns that order strings, 256 cases (3.5 s), and a fifth handcrafted column of ordered numbers. Left to the owner: the page could say a number's term equality is equality as an f64 |
faff23345 |
| 8 [Low] the note's net overstated what was already disclosed | fixed in the note (5455ae0d8). The long comment above still carries the old sentence. Read it as: the element index was already disclosed, but whether a node is a member or an element was disclosed only where selectors showed it, and is new for a node that is an only child in every row |
5455ae0d8 |
| 9 [Low] stale probe-comment text | fixed in mod.rs, the test's doc comment and the note |
981360faf, 5455ae0d8 |
| 10 [Nit] assertion messages, repeated banner | fixed | 981360faf, 7ca5b0df2 |
Now 38 distinguishers, 49 mutants, 11 leakage tests; every mutant is caught and UNNOTICED is still empty. The PR description's counts (35, 44, 9, 48 random cases) predate this.
Verification, from the worktree root on 294f03e50:
mise x --env test -- cargo nextest run -p stack-encrypt --all-features:888 tests run: 888 passed (1 slow), 1 skipped. That is 885, plus the two new harness tests, plus feat(stack-encrypt, eql): json_tree tokens derivable from the path, a self tag, client-side wildcards #1363's newjson_read_checkstest. The leakage suite alone: 11 passed in 12.6 s. The calibration is 12.6 s (was 9.6 s); the random positive control is 3.5 s.cargo test -p stack-encrypt --doc --all-features:ok. 50 passed.cargo clippy -p stack-encrypt --all-features --all-targets -- -D warnings: exit 0.cargo fmt --all --check: exit 0 at the root and insidepackages/eql.RUSTDOCFLAGS=-D warnings cargo doc -p stack-encrypt --all-features --no-deps: exit 0.npx -y @biomejs/biome@2.5.9 check --diagnostic-level=error: 992 files, no errors.packages/eql:mise run test:doc-anchors(123 links resolve),mise run docs:validate:source(passed).
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
cipherstash-agent
left a comment
cipherstash-agent
left a comment
There was a problem hiding this comment.
Review attestation — /code-review medium plus company-skills:cross-check-review-findings (loaded from cipherstash/skills), run against head 294f03e.
Critical / blocking: none.
- Test-only plus doc change.
cargo test -p stack-encrypt --all-features --test json_leakageat head: 11 passed. - 1 new finding (minor, inline): the leakage page says ordering terms "share leading bytes", but the CLLW layout only agrees to within ±1 per byte.
- Existing feedback (the author's 10-point review, fixed in
294f03e) was checked; nothing repeated here. The PR description's counts (35/44/9) are stale per the author's own note. - Cross-check was done inline, without verification subagents.
Generated by Claude Code
| | `x`, the self tag on every entry | Nothing on its own. It is a hash of the entry's own token salted by the document's `iv`, so it is unrelated bytes in every row and links no entry to another, and since every entry carries one, its presence marks none. Given a token, it is how the database finds that token's node in any row. | | ||
| | each ciphertext's length, and the number of entries | **The node count**, and for each node the length of its key name (for an object member) plus, for a scalar, the length of its value, exactly, less the AEAD's fixed overhead. A string's length in bytes and a number's length in digits are therefore visible per node. With the sealed root on, the root entry's length is the document's length as JSON. | | ||
| | `op`, the ordering term, only on number and string leaves at the paths the index orders | **The order of that leaf's value among every row that holds one, and which rows hold equal values.** The term is order-preserving encryption (CLLW OPE) under one key per column, so leaves at two ordered paths compare with each other too, and two ordered paths holding equal values carry equal terms. Strings are ordered and compared after their case and accents are folded, so `"Abc"`, `"abc"` and `"ábc"` get the same term: the equality it shows is of folded values. It is deterministic and not a random value: two values whose encodings share leading bytes (for a string, its first characters) get terms that share leading bytes, so the term also shows how long that common prefix is. **Its length is the leaf's length**: one byte per bit of the value's order encoding plus one, so 65 bytes for every number and `8 × (n + 1) + 1` bytes for a string whose folded UTF-8 form is `n` bytes. A term's presence at an ordered path also says the leaf is a number or a string. By default no path is ordered; see the options below. | | ||
| | `op`, the ordering term, only on number and string leaves at the paths the index orders | **The order of that leaf's value among every row that holds one, and which rows hold equal values.** The term is order-preserving encryption (CLLW OPE) under one key per column, so leaves at two ordered paths compare with each other too, and two ordered paths holding equal values carry equal terms. Strings are ordered and compared after their case and accents are folded, so `"Abc"`, `"abc"` and `"ábc"` get the same term: the equality it shows is of folded values. It is deterministic and not a random value: two values whose encodings share leading bits (for a number, its sign and then its exponent's leading bits; for a string, its first characters) get terms that share leading bytes, one byte per bit, so the term also shows how long that common prefix is, to the bit. So the stored terms alone sort the numbers at the ordered paths into signs and binary orders of magnitude, and show how far into their folded text two strings agree. **Its length is the leaf's length**: one byte per bit of the value's order encoding plus one, so 65 bytes for every number and `8 × (n + 1) + 1` bytes for a string whose folded UTF-8 form is `n` bytes. A term's presence at an ordered path also says the leaf is a number or a string. By default no path is ordered; see the options below. | |
There was a problem hiding this comment.
minor "terms that share leading bytes" is not what the term layout gives: chain::cllw_ope (packages/stack-encrypt/tests/json_leakage/chain.rs) adds a right-to-left carry, so bytes before the first differing bit agree only to within ±1 (mod 256), not exactly. Say "agree to within one" so the page matches the profile's ordering_term_prefixes reading.
Generated by Claude Code
…gs shows The profile refused a column whose ordered path held numbers in some rows and strings in others, and the page said nothing about one. The owner's decision: the page states it, and so does the profile. vitaminc_ore's order space salts its key with the domain, so a number's term and a string's are under different keys. A CLLW term begins with a carry byte and then the key's constant byte plus 128 for a first bit of 1. Under one key, then, every string compares the same way with every negative number and the same way with every non-negative one: the strings sort as one block, which the key places below the numbers, between the negative and non-negative ones, or above them. That is three placements, not the two of a single coin. Within each kind the order is the values'. The terms also show which leaves are numbers and which are strings. The profile now ranks classes within their kind, states each class's kind, and leaves the placement to the key (Leakage::kind_order). The simulator puts each kind's terms under a key of its own and draws key pairs until the strings fall at the placement asked. It refuses pairs whose front bytes tie to within a carry (about one in forty), under which a few pairs compare by later keyed bytes, a case no clause states. The real column's key picked one placement and the simulator cannot know which, so the calibration simulates each and counts worlds as told apart only when none agrees. The mutant mixed_kinds_jsonb_order fixes jsonb's numbers-first order. It is caught by ordering_term_order, ordering_matches and ordering_term_prefixes, because the test key puts the strings below the numbers; one world cannot tell a key that chose numbers-first from jsonb. A handcrafted column and a third kind of random column order numbers and strings at one path, with bounds of both kinds. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…sentence The owner's decision: the page says that a number's equality under its ordering term is as a 64-bit float. So two different integers past 2^53 can round to one float, share a term and tie under ORDER BY, while their value selectors differ. The profile already modelled this (the term is derived from as_f64), and the ordered-numbers column holds 9007199254740992 and 9007199254740993. What holds the sentence is a reading, NumberOrder::Exact, under which every distinct number gets its own term. It is a reading rather than a mutant in mutants.rs because a mutant transforms the leakage and never sees the values it would have to split. given_integers_ordered_exactly shows that ordering_term_order tells it from the real scheme on that column. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The owner's decision: the reworded prefix sentence stands, and the page says, after it, that this is CLLW's known profile and that a scheme which hides the prefix is planned for a later version. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The note's review section listed mixed kinds at an ordered path and f64 equality as left to the owner. Both are now decided, stated on the page and held by the profile, so the section says what was decided and how the test holds it, with the new totals. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The owner's three decisions, applied: head
|
…akage-profile Carries 8c843ee (the deepest-level container refusal) so #1362 holds #1363's final head before either merges. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Summary
Review pass (2026-10-25, head
294f03e50). After the port, a high-effort review found one real gap in the profile:Lomitted the ordering terms' shared prefixes. A CLLW term is one byte per bit of the value's order encoding, so two terms agree up to the first differing bit, and the stored terms alone sort numbers into signs and binary orders of magnitude and show how far two strings' folded text agrees. The page said this in prose; the profile, simulator and distinguishers now carry it (Leakage::ordering_prefixes,ordering_term_prefixes,ordering_prefixes_hidden). The review also added cross-kind equality and term-field distinguishers, four missing mutants, a wider corpus (big integers, floats, folding strings, nulls and containers at ordered paths, 256 random columns), and fixed stale prose. Owner's decisions (headfd229008d). The page now states what a path ordered over numbers and strings shows (the order between the kinds depends on the column key and is not jsonb's; within each kind it is the values'), that a number's equality under its term is as a 64-bit float, and that the shared prefix is CLLW's known profile, with a prefix-hiding scheme planned for a later version. The profile states the first as a clause (Leakage::kind_order, held by the mutantmixed_kinds_jsonb_order) and the second as a reading (NumberOrder::Exact, held bygiven_integers_ordered_exactly). Totals now: 38 distinguishers, 50 mutants, 12 tests; nextest 889 passed. The counts below are as the port left them where not updated.EQL v4's
json_treecolumn type stores an encrypted JSON document so PostgreSQL can search it without the key.packages/eql/docs/reference/json-tree-leakage.md(the "leakage page") says in prose what the database learns from the stored rows and from each query. This PR turns the page into a test: if the scheme reveals something the page leaves out, a test fails and names what gave it away.It writes the page's claims as code (a "profile"), builds a fake column and query log from those claims plus randomness alone (a "simulator"), and runs 38 checks an observer of the database could compute (the "distinguishers") on the fake column and on a real one sealed by Stack Encrypt. A calibration then drops each clause in turn (50 "mutant" profiles) and checks some distinguisher notices, so the test is known to be able to see what it claims to check.
This PR now targets #1363 and models its format. It was first written against the format before #1363 (one head token plus plaintext gates per read, member tokens carrying their ordinal, no self tag) and has been ported: every entry now carries a self tag
x, a member's token has no ordinal, and a subtree read sends the tokens of exactly the nodes it names, with wildcards expanded on the client to a bound and a probe per expanded array.docs/plans/2026-10-25-json-tree-leakage-before-and-after.mdcompares the two profiles clause by clause.Changes
packages/stack-encrypt/tests/json_leakage/(entrytests/json_leakage.rs):profile.rs: the leakage function over plaintext only. Every field is one clause, with the page sentence it encodes quoted in its doc comment. Every keyed value (selector, value selector, token, ordering term, self tag) is an opaque integer, equal exactly where the real bytes are. A read is its token list (each token named by its head and its member and element steps, in the order the client sends them), with the node each token names in each row, what it reaches below, whether it is a probe, and the budget.simulator.rs: builds wire-shaped rows and query terms from the profile output and a seeded RNG. It never sees plaintext or keys. It writes the real self tag on each node a sent token names and chains the parent and self tags below it with the real hash, as far as the profile says the token reaches; everywhere else the tags are random.chain.rs: the public hash functions (parent tag, self tag, member and element rules, and the database's test of a hit against the two rules).real.rsandobserve.rs: the real side through the public API (json_subtree_key_with_bound), and a parser of both worlds' wire JSON into one shape.distinguishers.rs: 38 exact features (35 at the port; the review pass addedordering_term_prefixes,cross_kind_equalitiesandterm_fields). Of the 27 the profile had, 26 are kept andgated_walk_resultsbecomessubtree_results(with the sealed-root budget fallback). Eight are new:stored_lengths,self_tag_equalities,token_counts,bound_recovery,token_hits,wildcard_positions,hit_rules,ancestor_links. One feature needs a probability bound (member_ordinals_follow_key_order,2⁻⁴⁰), written beside it.mutants.rs: 50 mutant profiles (44 at the port, 5 added by the review pass, 1 for the owner's mixed-kinds decision; 34 before: 28 kept, 4 redefined for token lists, the 2 gate mutants removed, 12 added, among them the pre-feat(stack-encrypt, eql): json_tree tokens derivable from the path, a self tag, client-side wildcards #1363 top-level token and member rule).corpus.rs: four handcrafted columns and a proptest generator of random columns, now with reads at bounds the rows overflow, two-level wildcards,$under both root options, a read past its budget, and strings that fold.calibration.rs: the tests.mod.rsholds the README-style documentation.packages/eql/docs/reference/json-tree-leakage.md: one sentence added (finding 1 below).docs/plans/2026-10-25-json-tree-leakage-before-and-after.md: the comparison of the two profiles.packages/stack-encrypt/Cargo.toml:proptestas a dev-dependency. feat(stack-encrypt, eql): json_tree tokens derivable from the path, a self tag, client-side wildcards #1363 also takes it; the merge keeps one line with one comment.Verification
All run from the worktree root on
09a9dfc0b(the merge of #1363 at997c64b1eplus the port). #1363 then moved tob992e787e, a one-line correction to the leakage page's v3 column; after merging it (561f273be) the leakage suite (9 passed) and Biome (992 files, no errors) were re-run, and nothing else, since it changes no code.mise x --env test -- cargo nextest run -p stack-encrypt --all-features:Summary [ 133.419s] 885 tests run: 885 passed (1 slow), 1 skipped. That is feat(stack-encrypt, eql): json_tree tokens derivable from the path, a self tag, client-side wildcards #1363's 876 and this PR's 9; the slow one is theuitrybuild test.mise x --env test -- cargo nextest run -p stack-encrypt --all-features --test json_leakage: 9 passed. The calibration is the slowest test, 9.6 s alone (12.0 s in the full parallel run); every other leakage test is under 0.5 s.cargo test -p stack-encrypt --doc --all-features:test result: ok. 50 passed; 0 failed.cargo clippy -p stack-encrypt --all-features --all-targets -- -D warnings: clean (exit 0).cargo fmt --all --check: exit 0 at the root and exit 0 insidepackages/eql.RUSTDOCFLAGS=-D warnings cargo doc -p stack-encrypt --all-features --no-deps: clean (exit 0).npx -y @biomejs/biome@2.5.9 check --diagnostic-level=error:Checked 992 files, no errors. No JSON fixture was added.pnpm run test:scripts(nothing underscripts/or.cargo/mutants.tomlchanged);cargo mutants(nothing undersrc/changed).json_tree.Calibration: which check catches each dropped clause
All 50 mutants are caught; none is listed as unnoticed. The positive control passes on the five handcrafted columns and on 256 random ones (48 at the port). The full table prints with
--no-captureonevery_clause_is_load_bearing; the comparison document lists every mutant and the clause it drops.Findings
Against the page as #1363 published it:
the_hit_rule_separates_it_from_the_real_schemeshows the sentence is needed (without it,hit_rules,token_reach,subtree_resultsandtoken_derivabilityseparate the worlds).Against the SQL's own comment, not the page (reported, not changed here):
json_tree_subtree's comment inpackages/eql/src/v3/json_tree/functions.sqlsays it cannot. It cannot from a token's bytes, but the probe is always the last token at its level, the count gives the bound away (65 tokens is one level atN = 64and nothing else), and a probe that finds something finds the element, whose selector differs from the leaf the other tokens find.a_read_gives_away_its_bound_and_its_probes.Against the profile itself:
"Carol"/"carol"and"ál"/"al", andgiven_strings_ordered_by_their_bytesholds the sentence.The three findings against the page before #1363 (an ordering term's length, equal values at two ordered paths, UTF-8 key order) are on the page now; each keeps a mutant or a reading the calibration catches. The "This page is checked by …" pointer is still not added to the page; that waits for finding 2 to be settled on #1363.
What the test shows #1363 changed, in short: at rest, only the self tag is new, and it reveals nothing (unlinkable across rows, distinct within one, a fixed 19 characters per entry). Under a query, the database now learns the token count (and with it the bound), which positions below the bound hold the rest of a wildcard path, which arrays are longer than the bound, the subtree of an element a probe finds, and member-or-element for every reached node. It no longer receives any ancestor's token (#1360: reading
$.profile.address.cityused to hand over$.profile's), the gates with their plaintext positions and selectors, or the head token. The stored order still reveals each row's shape (#1357).Related
Refs #1357, #1360, #1355, #1356, #1358, #1361, #1363.
Review notes
tests/json_leakage/mod.rs, thenprofile.rs, then the comparison document.mod.rsnow says so.mod.rs: the carries inside an ordering term (where two terms part is exact and is now a clause; which shared bytes differ by one is not); a near-tie between the number and string keys (about one key pair in forty), under which a few cross-kind pairs compare by later keyed bytes; the keyed member rank as a permutation (decidable only on wide objects, bound2⁻⁴⁰); the primitives; timing and the SQL'siandvfields;json_tree_childrenandjson_tree_child_count, which send one token a read of the same path sends; cross-column linkage.j. A stored order independent of shape, which is json_tree: the stored entry order reveals each row's shape and the alphabetical order of sibling keys #1357's goal, would scatter them. Both cannot hold as written.https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu