From 9d05f8b099381d605793ea3b367c978938bcfd3b Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 24 Oct 2026 18:26:39 +1100 Subject: [PATCH 1/5] fix(stack-encrypt): check that a json_tree read returns the entry asked for Every entry of a json_tree document is sealed under its own selector and position, and nothing in its AAD says which path a query asked for. The client opened what `->` and the subtree walks returned under the `s` and `j` that came back with each row, so a database could answer `$.salary` with the row's `$.bonus`, or with the sealed root (the whole document), or splice other entries of the row into a walk, and the client would rebuild whatever it was handed (#1355). A row now opens only through what the client derived for the read. `JsonSelector::open(node)` refuses an entry that does not carry the selector, before any key is retrieved. `JsonSubtreeKey::open(rows)` checks every row: a node a gated walk reaches carries the path's selector and the position the path ends in; every other row carries the selector of its parent's path extended by its decrypted label, which the key now derives through the keyset's index-key PRF it holds; and each subtree's rows are consecutive in the document, so a dropped, moved or orphaned row is refused. The refusal is a new `Error::JsonWrongEntry`: each entry is genuine, so it is not `Aead`, and it is not of a wrong shape, so it is not `OpenedType`. `JsonNode` no longer opens on its own. `eql-bindings` follows: an entry `->` returned opens through `JsonTreeExtracted::from_selector(&selector, entry)`, and `JsonTreeEntry` no longer opens on its own. Which row an entry comes from stays the database's word, by design (ADR 0010); the leakage page and the design record say so. Closes #1355 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- .../eql-json-tree-extracted-entry-check.md | 5 + docs/plans/2026-10-17-json-tree-index.md | 2 +- packages/eql/crates/eql-bindings/CHANGELOG.md | 7 +- .../crates/eql-bindings/src/v4/json_tree.rs | 60 +- .../eql/docs/reference/json-tree-leakage.md | 9 +- .../eql/tests/encryption/tests/json_tree.rs | 230 +++++++- packages/stack-encrypt/CHANGELOG.md | 18 +- packages/stack-encrypt/src/cipher.rs | 32 +- packages/stack-encrypt/src/codes.rs | 9 + packages/stack-encrypt/src/json/derive.rs | 432 ++++++++++++-- packages/stack-encrypt/src/json/mod.rs | 85 ++- packages/stack-encrypt/src/json/path.rs | 14 +- packages/stack-encrypt/src/json/query.rs | 54 +- packages/stack-encrypt/src/keyset.rs | 7 + packages/stack-encrypt/tests/json_index.rs | 125 +++-- .../stack-encrypt/tests/json_read_checks.rs | 531 ++++++++++++++++++ 16 files changed, 1489 insertions(+), 131 deletions(-) create mode 100644 .changeset/eql-json-tree-extracted-entry-check.md create mode 100644 packages/stack-encrypt/tests/json_read_checks.rs diff --git a/.changeset/eql-json-tree-extracted-entry-check.md b/.changeset/eql-json-tree-extracted-entry-check.md new file mode 100644 index 000000000..04662ee6c --- /dev/null +++ b/.changeset/eql-json-tree-extracted-entry-check.md @@ -0,0 +1,5 @@ +--- +"@cipherstash/eql": patch +--- + +EQL v4 `json_tree` (unreleased): the client now checks that what `->` and the subtree walks return is what it asked for. Every entry of a stored document is sealed under its own path selector and position, so any entry of the row opened, and a database could answer `doc -> s($.salary)` with the row's `$.bonus`, or with the sealed root (the whole document), or splice other entries into a walk. In the `eql-bindings` crate an entry `->` returned now opens through the selector that asked for it, `JsonTreeExtracted::from_selector(&selector, entry)`, which refuses an entry carrying any other selector; `JsonTreeEntry` no longer opens on its own. A walk's rows, opened through `JsonTreeSubtree::from_key(&key, rows)` as before, are each checked against the key's path: the node a gated walk reaches must carry the path's selector and position, every other row the selector of its parent's path and its own label, and the rows must follow each other in the document with no gap. A refusal is `stack_encrypt::Error::JsonWrongEntry`. Not checked, by design: which row of the column an entry comes from (ADR 0010). The SQL is unchanged. diff --git a/docs/plans/2026-10-17-json-tree-index.md b/docs/plans/2026-10-17-json-tree-index.md index 7c78baf30..92f2d37a2 100644 --- a/docs/plans/2026-10-17-json-tree-index.md +++ b/docs/plans/2026-10-17-json-tree-index.md @@ -496,7 +496,7 @@ Two of the three items that gate a release are closed here; the third, the final - **Planner fragility (gate 2).** `eql_v4.json_tree_query` is plpgsql with `COST 1000` so that a bitmap scan over the GIN index beats a sequential scan even when the documents are TOASTed and the heap looks like a few pages. That was observed in the benches, at 10,000 and 100,000 rows, on one PostgreSQL version, and nothing held it. `postgres_the_planner_picks_the_indexes_without_help` in the encryption suite now does: on a flat table (1,000 four-node documents, every row inline) and a nested table (300 documents of about 220 nodes, every row TOASTed, both regimes pinned by `pg_column_size`), with every planner setting at its default and after `ANALYZE`, containment plans as a bitmap scan over the GIN index and a range at an ordered path as a scan of the functional btree on `ord_term(doc -> selector)`. A control sets the function's cost to 1 for one transaction and shows the planner taking the sequential scan on the nested table, so the assertion is known to be able to fail. The suite runs in `test-eql.yml` in a new `rust-encryption-postgres` job, a matrix over the supported PostgreSQL versions (17 on a pull request, 14 to 17 on a push, like the SQLx jobs); it was a step of the single-version `rust-crates` job. Measured locally on 14, 15, 16 and 17 before the commit. - **Leakage (gate 3).** `packages/eql/docs/reference/json-tree-leakage.md` is the one page: what a stored row shows at rest, field by field; what each query type hands the database; what is authenticated; the side-by-side with v3 from section 6 above, brought up to date (no `g`, position-keyed `p`, ordering opt-in) and corrected where section 6 was wrong (the entry order; see the note under its table); and the two options that change the picture. It says what a token is and that it is column-wide and does not expire, which is the sentence a customer most needs before sending one. - **Open for the owner: the stored entry order (2026-10-23, review of #1336).** `sv` and `q` hold the nodes in the walk's order, depth-first, an object's members in the sorted order of their keys (serde_json's `Map` without `preserve_order`; a build that unifies that feature on would store insertion order instead, which leaks the application's field order). Across rows this reveals each row's shape, its parent relation, every array's length and the alphabetical order of sibling key names, which partly undoes what the keyed rank (decision 11) hides from a token holder. The leakage page says so. Closing it is a format change, not a documentation one: a stored order that does not follow the tree (a document-keyed permutation, say, with `j` the permuted position), a rebuild that does not read the tree from the order, and the same for the pairs of `q`, whose nesting would still group array elements. Not taken here. -- **Open for the owner: an extracted entry opens under the selector the server returns with it.** `JsonNode` and `JsonSubtree` open each entry under the `s` and `j` that arrive beside it, and under the data key the header that arrives with it names. The entry's AAD (`entry_aad` in `packages/stack-encrypt/src/json/derive.rs`) binds the keyset, the column's context, the data key's tag, the selector and the ordinal, and nothing that identifies a row. So a decrypted value is always one sealed in that column, but not necessarily the one at the path asked for, nor one from the row the result puts it in: `->` on `$.salary` can be answered with the row's `$.bonus`, with the sealed root (which opens to the whole document), or with another row's `$.salary` returned together with that row's header. The last is by design: two values sealed under the same context are interchangeable between rows (ADR 0010 in `packages/stack-encrypt/docs/adr`), and row provenance is the database's word. The leakage page states this under "What is authenticated". Checking the returned `s` against the selector the client derived would close the path substitution for `->`, though not the row one; a walk's rows would need each child's selector checked against its parent's path. +- **Taken (2026-10-24, #1355, PR PR_A): an extracted entry is checked to be the one asked for.** Until then `JsonNode` and `JsonSubtree` opened each entry under the `s` and `j` that arrived beside it, and under the data key the header that arrived with it named. The entry's AAD (`entry_aad` in `packages/stack-encrypt/src/json/derive.rs`) binds the keyset, the column's context, the data key's tag, the selector and the ordinal, and nothing that identifies a row. So a decrypted value was always one sealed in that column, but not necessarily the one at the path asked for, nor one from the row the result puts it in: `->` on `$.salary` could be answered with the row's `$.bonus`, with the sealed root (which opens to the whole document), or with another row's `$.salary` returned together with that row's header. Now a row opens only through what the client derived for the read. `JsonSelector::open(node)` refuses an entry whose `s` is not the selector (another path's entry, and the sealed root, whose selector is the root's, in place of a node) before any key is retrieved. `JsonSubtreeKey::open(rows)` refuses a row that does not fit the key's path: a node a gated walk reaches must carry the path's selector at depth `0`, and the position the path names when its last step is one; every other row must carry the selector of its parent's path extended by its own decrypted label (a key, or the wildcard for an element), which the key derives through the keyset's index-key PRF it now holds; and the rows of each subtree must be consecutive in the document, which is what a subtree is in a depth-first `sv`, so a row out of place, a dropped row and an orphan are refused. A reported depth `d` that disagrees is refused too; the rows are placed by their positions and child counts, not by `d`. The refusal is a new `Error::JsonWrongEntry`, not `Aead` (each entry is genuine) and not `OpenedType` (the entry is not of an unexpected shape; it answers another question). `eql-bindings` follows: `JsonTreeExtracted::from_selector(&selector, entry)` is how an entry `->` returned opens, and `JsonTreeEntry` no longer opens on its own. Still not covered: another row's `$.salary` with that row's header opens, by design (two values sealed under the same context are interchangeable between rows, ADR 0010 in `packages/stack-encrypt/docs/adr`, and row provenance is the database's word); and an array position before the last step of a path reads as the wildcard to every selector, so a walk's rows for `$.a[1].b` can be another element's `b` in the same row. Until the ungated walk returns the node it starts from (#1356), a walk with no gates is also not checked for which node at the path's selector it descends from, which matters only for a token chained through array positions (`$.l[3]`), whose children share every other element's selectors. ## Decisions to take before Phase 1 diff --git a/packages/eql/crates/eql-bindings/CHANGELOG.md b/packages/eql/crates/eql-bindings/CHANGELOG.md index 2622f6fa7..69ce58852 100644 --- a/packages/eql/crates/eql-bindings/CHANGELOG.md +++ b/packages/eql/crates/eql-bindings/CHANGELOG.md @@ -20,8 +20,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 opens at the path the walk's key was derived from: the rows are built with `JsonTreeSubtree::from_key(&key, rows)`, and there is no constructor without the key. + An entry `->` returned opens through the selector that asked for it, + `JsonTreeExtracted::from_selector(&selector, entry)`, which refuses an + entry carrying another selector (`stack_encrypt::Error::JsonWrongEntry`), + and a walk's rows through the key that keyed it, which checks each row + against the path the same way; `JsonTreeEntry` does not open on its own. The document is an `EncryptFrom` and `DecryptInto` target, - the entry and the subtree open through `DecryptInto`, and the needle and + the extracted entry and the subtree open through `DecryptInto`, and the needle and the ordering operand convert from Stack Encrypt's `JsonQuery` and `JsonLeafQuery`. `JsonTree::new(options)` is the `json_tree` layout under index options other than the defaults (a `Layout`, so a program diff --git a/packages/eql/crates/eql-bindings/src/v4/json_tree.rs b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs index 26507af85..ba0f77eb7 100644 --- a/packages/eql/crates/eql-bindings/src/v4/json_tree.rs +++ b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs @@ -23,8 +23,8 @@ use schemars::{schema_for, JsonSchema, Schema}; use serde::{Deserialize, Serialize}; use serde_json::Value; use stack_encrypt::json::{ - JsonDocument, JsonLeafQuery, JsonNode, JsonOptions, JsonPath, JsonQuery, JsonSubtree, - JsonSubtreeKey, + JsonDocument, JsonLeafQuery, JsonNode, JsonOptions, JsonPath, JsonQuery, JsonSelector, + JsonSubtree, JsonSubtreeKey, }; use stack_encrypt::target::{AeadContext, ExpectedContext, IndexSpec, Json, Layout}; use stack_encrypt::{ @@ -94,7 +94,13 @@ pub struct JsonTreeDocument { /// `c` the node's own ciphertext; `p` the parent tag below the chain's /// heads, which the walk probes by ordinal; `op` the leaf ordering term. /// `h`, `j`, `d`, `v` and `i` are what the extractors graft onto an entry -/// they return, so it opens on its own. +/// they return, so that it can be opened. +/// +/// An entry a query returned is the database's word and does not open on +/// its own: open the entry `->` returned through the selector that asked +/// for it ([`JsonTreeExtracted::from_selector`]), and a walk's rows through +/// the key that keyed it ([`JsonTreeSubtree::from_key`]). Each checks that +/// the rows are the ones it asked for. #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct JsonTreeEntry { @@ -165,6 +171,42 @@ pub struct JsonTreeLeafQuery { pub op: JsonTreeOrderingTerm, } +/// The entry `col -> selector` returned, with the selector that asked for +/// it: opens to the scalar the entry holds, or to the whole document for a +/// sealed root asked for at `$`. The only constructor, +/// [`from_selector`](Self::from_selector), takes the selector, and opening +/// refuses an entry that does not carry it +/// ([`Error::JsonWrongEntry`](stack_encrypt::Error::JsonWrongEntry)), as +/// [`JsonSelector::open`] does: a database that answers one path with +/// another path's entry, or with the sealed root, returns an entry that +/// would otherwise open. +#[derive(Clone, Debug, PartialEq)] +pub struct JsonTreeExtracted { + entry: JsonTreeEntry, + selector: JsonSelector, +} + +impl JsonTreeExtracted { + /// The entry `col -> selector` returned for `selector`, as the server + /// sent it. + pub fn from_selector(selector: &JsonSelector, entry: JsonTreeEntry) -> Self { + Self { + entry, + selector: *selector, + } + } + + /// The entry. + pub fn entry(&self) -> &JsonTreeEntry { + &self.entry + } + + /// The selector the read asked for. + pub fn selector(&self) -> &JsonSelector { + &self.selector + } +} + /// The rows a subtree walk returned (`eql_v4.json_tree_subtree` or /// `json_tree_subtree_at`), collected with the key that keyed the walk: /// opens to the value under the walked path, or to the list of complete @@ -396,14 +438,18 @@ where } /// An extracted entry opens to the scalar it holds, or to the whole -/// document for a sealed root, under the column it was extracted from. A +/// document for a sealed root asked for at `$`, under the column it was +/// extracted from, once it is shown to carry the selector asked for. A /// container node holds no value of its own; its subtree opens through /// [`JsonTreeSubtree`]. -impl DecryptInto for JsonTreeEntry { +impl DecryptInto for JsonTreeExtracted { type Context = NonEmpty; fn decryption(self, context: Self::Context) -> Decryption { - match self.node() { - Ok(node) => node.decryption(AeadContext::from(context)), + match self.entry.node() { + Ok(node) => self + .selector + .open(node) + .decryption(AeadContext::from(context)), Err(error) => Decryption::failed(error), } } diff --git a/packages/eql/docs/reference/json-tree-leakage.md b/packages/eql/docs/reference/json-tree-leakage.md index bfb6ffab2..265aa6fd6 100644 --- a/packages/eql/docs/reference/json-tree-leakage.md +++ b/packages/eql/docs/reference/json-tree-leakage.md @@ -39,7 +39,12 @@ A query hands the database a term derived from the plaintext it asks about, and Only the ciphertext `c`. Opening an entry verifies that it was sealed under this keyset, for this column, under the data key its row's header names (a fresh one for every row), with the selector `s` and the position `j` it is opened with; a ciphertext moved into another row or column, or opened under another selector or position, fails. Nothing in it names a row, so an entry that travels with its own row's header still opens; see below. When the client opens a whole document it takes each entry's position from its place in `sv`, so a moved, swapped or re-labelled entry fails to open. The selectors, the projection, the parent tags, the ordering terms, the header's sealed-root flag, and which entries a row holds are not authenticated, by the same rule every EQL stored value follows (ADR 0010 in `packages/stack-encrypt/docs/adr`). -The consequence to design around: tampering with those fields changes **which rows a query returns, where a walk stops, and which entries of a row an extraction or a walk hands back**. An entry the database returns from `->` or a walk carries its own `s` and `j`, and the client opens it under those, and the header it carries names the data key, so a decrypted value is always one that was sealed in this column, but the client does not check today that it is the entry at the path it asked for, or that it comes from the row the result puts it in: a database that wants to can answer `$.salary` with the same row's `$.bonus`, or with another row's `$.salary`. A query's result is not integrity-protected; each decrypted value is genuine, but where it came from is the database's word. +The consequence to design around: tampering with those fields changes **which rows a query returns, where a walk stops, and which entries of a row an extraction or a walk hands back**. An entry the database returns from `->` or a walk carries its own `s` and `j`, and the header it carries names the data key, so a decrypted value is always one that was sealed in this column. The client also checks that it is the entry it asked for, from what it derived for the query, and refuses anything else (`JsonWrongEntry`): + +- **`->`.** The entry must carry the selector the client derived for the path. Another path's entry, such as the row's `$.bonus` returned for `$.salary`, is refused, and so is the sealed root returned for a field (it opens to the whole document); the root is accepted only when `$` itself was asked for. +- **A walk.** A node a gated walk reaches must carry the path's selector, and the position the path ends in, if it ends in one. Every other row must carry the selector of its parent's path extended by its own key (or the wildcard, for an array element), which the client derives from the decrypted key, and a subtree's rows must follow each other in the document with no gap. A row of another path, a dropped or reordered row, and a row whose parent is not among the rows are refused. + +What the checks do not cover: **which row an entry comes from**. Nothing in an entry names its row, so another row's `$.salary`, returned with that row's header, opens; that is the rule every EQL stored value follows (ADR 0010). And **a position before the last step of a path**: every element of an array shares its selector, so a walk for `$.orders[1].items` can be answered with `$.orders[0].items` of the same row. A query's result is not integrity-protected as a whole: each decrypted value is genuine and at the path asked for, but which row it came from is the database's word. ## Compared with EQL v3 (`public.eql_v3_json_search`) @@ -62,6 +67,6 @@ The consequence to design around: tampering with those fields changes **which ro - **`sealed_root`** (default on). The root entry's ciphertext is the whole document, so a full read is one decryption and the broadest token a query can send is one top-level field's. The costs are that the root entry's length reveals the document's size, the row is larger, and the root's direct members are marked by their missing `p`, which puts each top-level field's node count in plain view. Off, there is no document-sized ciphertext, but every subtree read whose path begins with a key sends the root's token, the whole column's shape. - **`ordered`** (default none). Each path listed carries an ordering term and takes on the `op` row above; a path not listed cannot be range-queried or sorted, and a query that tries is refused by the client before any key loads. Order only the paths your queries sort or range over. -Two disclosures above are properties of the stored format that no option changes today: the depth-first entry order, with members in sorted key order, and the client opening an extracted entry under the selector the database returns with it. Both are open decisions recorded in the design document (`docs/plans/2026-10-17-json-tree-index.md` at the repository root, Phase 4 notes). +One disclosure above is a property of the stored format that no option changes today: the depth-first entry order, with members in sorted key order. It is an open decision recorded in the design document (`docs/plans/2026-10-17-json-tree-index.md` at the repository root, Phase 4 notes) and in #1357. To report a concern with any of this, see `SECURITY.md` at the repository root. diff --git a/packages/eql/tests/encryption/tests/json_tree.rs b/packages/eql/tests/encryption/tests/json_tree.rs index e1c4a97aa..ee9d0fb8e 100644 --- a/packages/eql/tests/encryption/tests/json_tree.rs +++ b/packages/eql/tests/encryption/tests/json_tree.rs @@ -11,11 +11,12 @@ mod common; use eql_bindings::v4::json_tree::{ - JsonTree, JsonTreeDocument, JsonTreeEntry, JsonTreeLeafQuery, JsonTreeQuery, JsonTreeSubtree, + JsonTree, JsonTreeDocument, JsonTreeEntry, JsonTreeExtracted, JsonTreeLeafQuery, JsonTreeQuery, + JsonTreeSubtree, }; use eql_bindings::Identifier; use serde_json::{json, Map, Value}; -use stack_encrypt::json::{JsonDocument, JsonOptions, JsonPath, JsonSubtreeKey}; +use stack_encrypt::json::{JsonDocument, JsonOptions, JsonPath, JsonSelector, JsonSubtreeKey}; use stack_encrypt::plan::{Laid, ValuePlan}; use stack_encrypt::target::{Json, Layout}; use stack_encrypt::{op, NonEmpty, Plan}; @@ -118,16 +119,49 @@ async fn a_document_round_trips_through_the_typed_target() { ); // An entry lifted from the stored document, with the header and its - // ordinal grafted as `->` grafts them, opens to its scalar. + // ordinal grafted as `->` grafts them, opens to its scalar through the + // selector that asked for it, and only through that one. + let keyset = cipher.default_keyset(); + let selector = |path: &str| { + let path = JsonPath::parse(path).unwrap(); + let keyset = keyset.clone(); + async move { keyset.json_selector(&path, column()).await.unwrap() } + }; + let hello = selector("$.hello").await; let mut entry = stored.sv[1].clone(); entry.h = Some(stored.h.clone()); entry.j = Some(1); entry.i = Some(stored.i.clone()); entry.v = Some(stored.v); - let scalar: Value = cipher.decrypt_as(entry, column()).await.unwrap(); + let scalar: Value = cipher + .decrypt_as( + JsonTreeExtracted::from_selector(&hello, entry.clone()), + column(), + ) + .await + .unwrap(); assert_eq!(scalar, json!("world-1")); + let asked = JsonTreeExtracted::from_selector(&hello, entry.clone()); + assert_eq!(asked.entry(), &entry); + assert_eq!(asked.selector(), &hello); + let number = selector("$.number").await; + let refused = cipher + .decrypt_as::(JsonTreeExtracted::from_selector(&number, entry), column()) + .await; + assert!( + matches!( + refused, + Err(stack_encrypt::Error::JsonWrongEntry { + expected: "the entry at the path asked for" + }) + ), + "another path's selector refuses the entry: {refused:?}" + ); let bare = stored.sv[1].clone(); - assert!(cipher.decrypt_as::(bare, column()).await.is_err()); + assert!(cipher + .decrypt_as::(JsonTreeExtracted::from_selector(&hello, bare), column()) + .await + .is_err()); } #[tokio::test] @@ -471,14 +505,14 @@ fn postgres_extracts_walks_and_orders_by_path() { ) .unwrap(); } - let selector = |path: &str| { + let selector_of = |path: &str| -> JsonSelector { wait( &runtime, keyset.json_selector(&JsonPath::parse(path).unwrap(), column()), ) .unwrap() - .to_string() }; + let selector = |path: &str| selector_of(path).to_string(); let subtree_key = |path: &str| { wait( &runtime, @@ -493,8 +527,13 @@ fn postgres_extracts_walks_and_orders_by_path() { let entry = |row: &postgres::Row| -> JsonTreeEntry { serde_json::from_str(&row.get::<_, String>(0)).unwrap() }; - let open = - |e: JsonTreeEntry| -> Value { wait(&runtime, cipher.decrypt_as(e, column())).unwrap() }; + let open = |asked: &JsonSelector, e: JsonTreeEntry| -> Value { + wait( + &runtime, + cipher.decrypt_as(JsonTreeExtracted::from_selector(asked, e), column()), + ) + .unwrap() + }; // `->` by selector opens to the leaf; a missing path is NULL. let hello = selector("$.hello"); @@ -504,7 +543,7 @@ fn postgres_extracts_walks_and_orders_by_path() { &[&hello], ) .unwrap(); - assert_eq!(open(entry(&row)), json!("world-2")); + assert_eq!(open(&selector_of("$.hello"), entry(&row)), json!("world-2")); let missing = db .query_one( "SELECT (doc -> $1::text)::jsonb::text FROM json_tree_rows WHERE id = 3", @@ -532,7 +571,7 @@ fn postgres_extracts_walks_and_orders_by_path() { ) .unwrap() .iter() - .map(|r| open(entry(r))) + .map(|r| open(&selector_of("$.tags[*]"), entry(r))) .collect(); assert_eq!(tags, vec![json!("x"), json!("y")]); @@ -1313,6 +1352,175 @@ fn postgres_a_position_reaches_array_elements_only() { ); } +/// Needs an EQL v4 installation. What `->` and the walks return is checked +/// by the client to be what it asked for: every entry of a row is genuine +/// and opens under its own selector and position, so an answer made of the +/// wrong entries would open unless the client refuses it. Real rows come +/// back from PostgreSQL here, from a database answering the wrong question +/// (another path's entry, the sealed root, another position's subtree) and +/// from stored rows tampered with (a parent tag rewritten so the walk takes +/// a top-level field for a member, and two entries swapped in `sv`). Each +/// is refused: the wrong entry as `JsonWrongEntry`, a moved entry as +/// `Aead`, since its ciphertext is bound to the position it was sealed at. +#[test] +#[ignore = "requires EQL_TEST_DATABASE_URL pointing to an EQL v4 database"] +fn postgres_hostile_answers_are_refused_by_the_client() { + use stack_encrypt::Error; + + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let (cipher, _) = wait(&runtime, common::cipher()); + let keyset = cipher.default_keyset(); + let mut db = connect(); + db.batch_execute( + "CREATE TEMP TABLE json_tree_hostile (id int PRIMARY KEY, doc public.eql_v4_json_tree NOT NULL)", + ) + .unwrap(); + for id in [1, 2] { + let stored = serde_json::to_string(&wait(&runtime, seal(&cipher, &fixture()))).unwrap(); + db.execute( + "INSERT INTO json_tree_hostile VALUES ($1, $2::text::jsonb::public.eql_v4_json_tree)", + &[&id, &stored], + ) + .unwrap(); + } + let selector_of = |path: &str| -> JsonSelector { + wait( + &runtime, + keyset.json_selector(&JsonPath::parse(path).unwrap(), column()), + ) + .unwrap() + }; + let subtree_key = |path: &str| -> JsonSubtreeKey { + wait( + &runtime, + keyset.json_subtree_key( + &JsonPath::parse(path).unwrap(), + &JsonOptions::default(), + column(), + ), + ) + .unwrap() + }; + // `doc -> selector` in row `id`. + let arrow = |db: &mut postgres::Client, id: i32, selector: &JsonSelector| -> JsonTreeEntry { + let row = db + .query_one( + "SELECT (doc -> $1::text)::jsonb::text FROM json_tree_hostile WHERE id = $2", + &[&selector.to_string(), &id], + ) + .unwrap(); + serde_json::from_str(&row.get::<_, String>(0)).unwrap() + }; + // The walk `key` keys in row `id`, walked out (budget 0). + let walked = |db: &mut postgres::Client, id: i32, key: &JsonSubtreeKey| -> Vec { + db.query( + "SELECT e::jsonb::text FROM json_tree_hostile, eql_v4.json_tree_subtree_at(doc, $1::text, $2::text[], 0) e WHERE id = $3", + &[&key.token(), &key.gates(), &id], + ) + .unwrap() + .iter() + .map(|r| serde_json::from_str(&r.get::<_, String>(0)).unwrap()) + .collect() + }; + let extracted = |asked: &JsonSelector, entry: JsonTreeEntry| -> Result { + wait( + &runtime, + cipher.decrypt_as(JsonTreeExtracted::from_selector(asked, entry), column()), + ) + }; + let opened = |key: &JsonSubtreeKey, rows: Vec| -> Result { + wait( + &runtime, + cipher.decrypt_as(JsonTreeSubtree::from_key(key, rows), column()), + ) + }; + let wrong = |result: Result, what: &str| { + assert!( + matches!(result, Err(Error::JsonWrongEntry { .. })), + "{what}: {result:?}" + ); + }; + + // `->` on `$.hello` answered with the row's `$.number`, and with the + // sealed root, which opens to the whole document. + let hello = selector_of("$.hello"); + assert_eq!( + extracted(&hello, arrow(&mut db, 1, &hello)).unwrap(), + json!("world-1") + ); + wrong( + extracted(&hello, arrow(&mut db, 1, &selector_of("$.number"))), + "another path's entry", + ); + let root = arrow(&mut db, 1, &selector_of("$")); + wrong(extracted(&hello, root.clone()), "the sealed root"); + assert_eq!( + extracted(&selector_of("$"), root).unwrap(), + fixture(), + "the root asked for at `$`" + ); + + // The walk to `$.nested.list[1]` answered with the rows of the walk to + // `$.nested.list[0]`: the same selectors, another position. + let second = subtree_key("$.nested.list[1]"); + assert_eq!( + opened(&second, walked(&mut db, 1, &second)).unwrap(), + json!({"x": 2}) + ); + wrong( + opened( + &second, + walked(&mut db, 1, &subtree_key("$.nested.list[0]")), + ), + "another position's subtree", + ); + + // A stored row whose `$.number` (entry 9) carries a parent tag under + // `$.nested`'s token, at the ordinal after its two members: the walk + // takes it for nested's third member, and returns it right after + // nested's subtree, where it sits in `sv`. + let nested = subtree_key("$.nested"); + assert!(nested.gates().is_empty()); + assert_eq!( + opened(&nested, walked(&mut db, 1, &nested)).unwrap(), + fixture()["nested"] + ); + db.execute( + "UPDATE json_tree_hostile SET doc = jsonb_set(doc::jsonb, '{sv,9,p}', + to_jsonb(eql_v4_internal.json_tree_parent_tag( + eql_v4_internal.json_tree_iv(doc::jsonb ->> 'h'), decode($1::text, 'base64'), 2)))::public.eql_v4_json_tree + WHERE id = 1", + &[&nested.token()], + ) + .unwrap(); + let rows = walked(&mut db, 1, &nested); + assert_eq!(rows.len(), 7, "the walk took number for a member"); + assert_eq!( + rows.last().map(|e| (e.j, e.d)), + Some((Some(9), Some(1))), + "number, one level below nested" + ); + wrong(opened(&nested, rows), "a top-level field as a member"); + + // A stored row whose `deep` (3) and `number` (9) swapped places: each + // entry is found where it now sits, and fails to open there. + db.execute( + "UPDATE json_tree_hostile SET doc = jsonb_set(jsonb_set(doc::jsonb, + '{sv,3}', doc::jsonb -> 'sv' -> 9), '{sv,9}', doc::jsonb -> 'sv' -> 3)::public.eql_v4_json_tree + WHERE id = 2", + &[], + ) + .unwrap(); + let deep = selector_of("$.nested.deep"); + let moved = extracted(&deep, arrow(&mut db, 2, &deep)); + assert!(matches!(moved, Err(Error::Aead)), "`->`: {moved:?}"); + let moved = opened(&nested, walked(&mut db, 2, &nested)); + assert!(matches!(moved, Err(Error::Aead)), "the walk: {moved:?}"); +} + /// Needs an EQL v4 installation. A stored document is untrusted input: its /// parent tags are not authenticated, so whoever holds a token can write a /// chain of them deeper than any sealed document. A child's token is a diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index f2ce6411d..409199918 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -21,8 +21,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `EncryptFrom` / `DecryptInto` target, and it answers `op::Contains`, `op::ContainedBy`, `op::AtPath(op::Eq)` (a scalar at a path) and the orderings at an ordered path; `KeysetCipher::json_selector` - and `json_subtree_key` derive the extraction keys, and `JsonNode` and - `JsonSubtree` open what the server returns. `JsonOptions::sealed_root`, + and `json_subtree_key` derive the extraction keys. What the server + returns (`JsonNode` rows) opens only through what the client derived for + the read, which checks that the rows are the ones it asked for: an + entry's ciphertext is bound to its own selector and position, so any + entry of the row would otherwise open. `JsonSelector::open(node)` builds + a `JsonExtracted`, which refuses an entry that does not carry the + selector (another path's, or the sealed root in place of a node) before + any key is retrieved; `JsonSubtreeKey::open(rows)` builds a + `JsonSubtree`, which refuses a row that is not where the walk's rows must + be: a reached node without the path's selector or position, a row + without the selector of its parent's path and its own label (the key + holds the keyset's index-key PRF and the column's context to derive + it), and a row that does not follow the one before it in the document + or whose parent is not among the rows. Each refusal is the new + `Error::JsonWrongEntry`. Not checked, by design, is which row of the + column an entry comes from (ADR 0010). `JsonOptions::sealed_root`, on by default, makes the root entry the whole document so no token for the root exists. A document is at most 127 levels deep. New `TermError` variants: `JsonPath`, `JsonTooDeep`, `JsonTooWide`, `JsonSealedRoot`, diff --git a/packages/stack-encrypt/src/cipher.rs b/packages/stack-encrypt/src/cipher.rs index 62c79fa50..3ebe1d352 100644 --- a/packages/stack-encrypt/src/cipher.rs +++ b/packages/stack-encrypt/src/cipher.rs @@ -173,6 +173,34 @@ pub enum Error { help("Open the value as the type it was sealed as. The ciphertext is authentic: it was sealed as another type, not altered.") )] OpenedType { expected: &'static str }, + /// The database answered a JSON index read with an entry other than + /// the one the read asked for: `->` returned the entry of another path, + /// or the sealed root in place of a node, or a subtree walk returned a + /// row that is not where the walk's key says its rows are (a node of + /// another path, a row out of its place in the document, or a row + /// whose parent is not among the rows). Refused by the client, which + /// derived the path's selector itself; see + /// [`JsonSelector::open`](crate::json::JsonSelector::open) and + /// [`JsonSubtreeKey::open`](crate::json::JsonSubtreeKey::open). + /// + /// Not [`Aead`](Self::Aead): every entry may be genuine, sealed in + /// this column and this row. Not [`OpenedType`](Self::OpenedType) + /// either: the entry is not of an unexpected shape, it is the answer to + /// another question. A database that answers the query it was sent + /// never returns one, so this says the rows were tampered with or the + /// query that produced them was not the one the key or selector keyed. + /// + /// Which row a value belongs to is not checked: an entry from another + /// row of the same column at the same path, returned with that row's + /// header, opens (ADR 0010, `docs/adr/0010-what-a-stored-value-authenticates.md`). + /// + /// `expected` names what the read expected, never what the entry held. + #[error("the database returned a JSON index entry other than {expected}")] + #[diagnostic( + code(stack_encrypt::json_wrong_entry), + help("The entry opened, but it is not the one the query asked for. A database that answers the query it was sent never returns one: check that the rows came from the query the selector or subtree key was derived for, and treat them as tampered with if they did.") + )] + JsonWrongEntry { expected: &'static str }, /// ZeroKMS returned a different number of keys than were requested. #[error("expected {expected} data keys from ZeroKMS but received {received}")] #[diagnostic(code(stack_encrypt::key_count_mismatch))] @@ -343,7 +371,9 @@ impl crate::ErrorPayload for Error { ("expected", (*expected).into()), ("received", (*received).into()), ]), - Self::OpenedType { expected } => payload([("expected", (*expected).into())]), + Self::OpenedType { expected } | Self::JsonWrongEntry { expected } => { + payload([("expected", (*expected).into())]) + } Self::DescriptorTooLong { len } => payload([ ("len", (*len).into()), ("limit", Descriptor::MAX_LEN.into()), diff --git a/packages/stack-encrypt/src/codes.rs b/packages/stack-encrypt/src/codes.rs index f731ed26c..0d3836fc9 100644 --- a/packages/stack-encrypt/src/codes.rs +++ b/packages/stack-encrypt/src/codes.rs @@ -49,6 +49,9 @@ mod tests { Error::Kms(_) => Error::Kms(crate::kms::Error::Unexpected("kms".into())), Error::Aead => Error::Aead, Error::OpenedType { .. } => Error::OpenedType { expected: "i32" }, + Error::JsonWrongEntry { .. } => Error::JsonWrongEntry { + expected: "the entry at the path asked for", + }, Error::KeyCountMismatch { .. } => Error::KeyCountMismatch { expected: 2, received: 1, @@ -397,6 +400,12 @@ mod tests { Box::new(Error::OpenedType { expected: "i32" }), json!({ "expected": "i32" }), ), + ( + Box::new(Error::JsonWrongEntry { + expected: "the entry at the path asked for", + }), + json!({ "expected": "the entry at the path asked for" }), + ), ( Box::new(LeafBytesError::UnknownVersion(9)), json!({ "version": 9 }), diff --git a/packages/stack-encrypt/src/json/derive.rs b/packages/stack-encrypt/src/json/derive.rs index 17900588f..5af1160dc 100644 --- a/packages/stack-encrypt/src/json/derive.rs +++ b/packages/stack-encrypt/src/json/derive.rs @@ -8,6 +8,7 @@ //! ordinal `j` is its place in that order, and the AAD of its ciphertext //! binds it there. use std::collections::BTreeMap; +use std::sync::Arc; use serde_json::{Map, Number, Value}; use sha2::{Digest, Sha256}; @@ -16,15 +17,16 @@ use vitaminc_aead::{Cipher, Context, IntoAad, IntoContext, LocalCipherText}; use vitaminc_encrypt::{Aes256Cipher, Key as AesKey}; use vitaminc_hmac::HmacSha256Prf; use vitaminc_prf::{BlockVisitor, IntoPrfContext, PrfValue}; -use vitaminc_protected::{Controlled, Protected}; +use vitaminc_protected::{AsProtectedRef, Controlled, Protected}; -use super::path::{JsonPath, Segment}; +use super::path::{key_step_len, push_key, JsonPath, Segment, ARRAY_STEP}; use super::plain::{ insert_member, protected_array, protected_object, push_element, read_protected, take_protected, PlainJson, }; use super::query::{Gate, JsonSelector, JsonSubtreeKey}; -use super::{JsonDocument, JsonEntry, JsonHeader, JsonNode, JsonOptions, JsonSubtree, Tag}; +use super::{JsonDocument, JsonEntry, JsonExtracted, JsonHeader, JsonOptions, JsonSubtree, Tag}; +use crate::keyset::KeysetState; use crate::sem::TermError; use crate::target::{AeadContext, CallerContext, Decryption, Pending, Request}; use crate::{DecryptInto, Descriptor, Error, KeysetCipher, NonEmpty, StackCipher}; @@ -268,6 +270,18 @@ fn prf16( domain: &[u8], context: &Context<'_>, input: &[u8], +) -> Result { + prf16_of(prf, domain, context, input) +} + +/// [`prf16`] over any PRF input: a `Protected>` derives exactly +/// what its bytes do, so input built from decrypted plaintext stays +/// wrapped up to the PRF. +fn prf16_of( + prf: &HmacSha256Prf, + domain: &[u8], + context: &Context<'_>, + input: impl PrfValue, ) -> Result { let block: [u8; 32] = input .prf_visit_with_context( @@ -280,6 +294,53 @@ fn prf16( Ok(truncate(&block)) } +/// What derives path selectors under one column after the keyset handle +/// that derived a subtree key is gone: the keyset's loaded state (its +/// index-key PRF), shared, and the column's PRF context. A +/// [`JsonSubtreeKey`] holds one, so that opening a walk's rows can derive +/// the selector each row must carry from its parent's path and its own +/// decrypted label. Never printed; compared by keyset and context. +#[derive(Clone)] +pub(crate) struct Selectors { + keyset: Arc, + context: Context<'static>, +} + +impl Selectors { + pub(crate) fn new(keyset: Arc, context: Context<'static>) -> Self { + Self { keyset, context } + } + + /// The selector of the path whose selector bytes are `bytes`. + fn of(&self, bytes: Protected>) -> Result { + prf16_of(&self.keyset.prf, SELECTOR_DOMAIN, &self.context, bytes).map_err(Error::from) + } +} + +#[cfg(test)] +impl Selectors { + /// Selectors under a fixed test keyset (index key `[7; 32]`, as the + /// unit tests' `prf()`) and the column `users/profile`. + pub(crate) fn for_tests() -> Self { + use vitaminc_prf::PrfKeyInit; + Self::new( + Arc::new(KeysetState { + id: Uuid::from_u128(1), + name: None, + prf: HmacSha256Prf::new(Protected::new([7; 32])), + }), + Context::from_encoded(b"users/profile").into_owned(), + ) + } +} + +impl PartialEq for Selectors { + fn eq(&self, other: &Self) -> bool { + self.keyset.id == other.keyset.id && self.context == other.context + } +} +impl Eq for Selectors {} + /// The first sixteen bytes of SHA-256 over the concatenation of `parts`. /// Plain SHA-256 by design: the design's "Why plain SHA-256 is enough here" /// records why length extension does not apply. @@ -1076,22 +1137,35 @@ impl DecryptInto for JsonDocument { } } -/// Open one extracted node: the scalar it holds, or the whole document for -/// a sealed root. A container node holds no value of its own and is -/// refused with [`Error::OpenedType`]; its subtree opens through -/// [`JsonSubtree`]. -impl DecryptInto for JsonNode { +/// The refusal of a row that is not the one a read asked for. +fn wrong(expected: &'static str) -> Error { + Error::JsonWrongEntry { expected } +} + +/// Open the entry `->` returned for a selector: the scalar it holds, or the +/// whole document for a sealed root asked for at `$`. An entry that does +/// not carry the selector asked for is refused with +/// [`Error::JsonWrongEntry`] before any key is retrieved: it is another +/// path's entry, or the sealed root (whose selector is the root's) in place +/// of a node, and it would open, since every entry is sealed under its own +/// selector. A container node holds no value of its own and is refused +/// with [`Error::OpenedType`]; its subtree opens through [`JsonSubtree`]. +impl DecryptInto for JsonExtracted { type Context = AeadContext; fn decryption(self, context: Self::Context) -> Decryption { + let (node, selector) = self.into_parts(); + if node.entry.s != selector { + return Decryption::failed(wrong("the entry at the path asked for")); + } Decryption::from_open(move |cipher| { - let (request, aad) = match retrieval(&self.header, context) { + let (request, aad) = match retrieval(&node.header, context) { Ok(parts) => parts, Err(error) => return Pending::failed(cipher, error), }; Pending::request(cipher, vec![request], move |responses| { let key = responses.next_retrieved_key()?; let aes = leaf_cipher(&key)?; - let opened = open_entry(&aes, &self.header, &aad, &self.entry, self.j)?; + let opened = open_entry(&aes, &node.header, &aad, &node.entry, node.j)?; opened .whole .or(opened.scalar) @@ -1113,13 +1187,15 @@ impl DecryptInto for JsonNode { /// (`json_tree_subtree_at`); the children of the one node at the path, for /// a key without (`json_tree_subtree`). An object's members come in an /// order neither form fixes, and without their keys; open it as a -/// [`Value`] to keep them. +/// [`Value`] to keep them. Every row is checked against the key first +/// ([`JsonSubtreeKey::open`]); a row that does not fit is +/// [`Error::JsonWrongEntry`]. impl DecryptInto> for JsonSubtree { type Context = AeadContext; fn decryption(self, context: Self::Context) -> Decryption, K> { Decryption::from_open(move |cipher| { if self.is_sealed_root() { - let below = self.below; + let below = self.below(); return open_fallback(self, cipher, context).try_map(move |(path, mut whole)| { // The selected nodes move out of the document's // `Protected` into one of their own, leaving `null` @@ -1163,7 +1239,9 @@ impl DecryptInto> for JsonSubtree { /// descendants, and `->` on the node's selector returns the node itself. /// The sealed root a walk fell back to opens whole, and the key's path /// ([`JsonSubtreeKey::open`]) selects the one value under it; a path that -/// reaches no node, or more than one, is [`Error::OpenedType`]. +/// reaches no node, or more than one, is [`Error::OpenedType`]. Every row +/// is checked against the key first ([`JsonSubtreeKey::open`]); a row that +/// does not fit is [`Error::JsonWrongEntry`]. impl DecryptInto for JsonSubtree { type Context = AeadContext; fn decryption(self, context: Self::Context) -> Decryption { @@ -1191,9 +1269,9 @@ impl DecryptInto for JsonSubtree { } }); } - let reached = self.nodes.iter().any(|n| n.depth == Some(0)); + let below = self.below(); open_subtree(self, cipher, context).try_map(move |forest| { - if reached { + if !below { let mut values: Vec> = forest.into_iter().map(|(_, v)| v).collect(); return match values.len() { @@ -1230,14 +1308,27 @@ impl DecryptInto for JsonSubtree { } } +/// One row of a walk, opened: the node, and the selector, position and +/// depth the server returned it with. The depth is the server's report and +/// optional on the wire; the rows are placed by their positions and the +/// opened nodes' child counts, and a depth that disagrees with the place +/// a row is put in is refused. +struct Row { + node: Opened, + s: Tag, + j: u64, + d: Option, +} + /// Open every row and rebuild each complete subtree, labelled, each still -/// under its `Protected`. +/// under its `Protected`, checking every row against the key as it goes +/// ([`place`]). fn open_subtree<'a, K>( subtree: JsonSubtree, cipher: &'a StackCipher, context: AeadContext, ) -> Pending<'a, Vec<(Label, Protected)>, K> { - let mut nodes = subtree.nodes; + let (mut nodes, key) = subtree.into_parts(); nodes.sort_by_key(|n| n.j); // A walk returns each entry once. Two rows at one ordinal are the // server repeating an entry, which would let it line up one container @@ -1257,23 +1348,198 @@ fn open_subtree<'a, K>( Err(error) => return Pending::failed(cipher, error), }; Pending::request(cipher, vec![request], move |responses| { - let key = responses.next_retrieved_key()?; - let aes = leaf_cipher(&key)?; - let opened = nodes + let key_bytes = responses.next_retrieved_key()?; + let aes = leaf_cipher(&key_bytes)?; + let rows = nodes .iter() - .map(|n| open_entry(&aes, &header, &aad, &n.entry, n.j)) - .collect::, _>>()?; - let mut forest = Vec::new(); - let mut rest = opened.into_iter(); - while let Some(top) = rest.next() { - let (label, top) = take_label(top); - let value = rebuild(top, &mut rest, 0)?; - forest.push((label, value)); - } - Ok(forest) + .map(|n| { + Ok(Row { + node: open_entry(&aes, &header, &aad, &n.entry, n.j)?, + s: n.entry.s, + j: n.j, + d: n.depth, + }) + }) + .collect::, Error>>()?; + place(rows, &key) }) } +/// Put the opened rows of a walk back into the subtrees they hold, in +/// document order, refusing any row that is not where the key says the +/// walk's rows are ([`Error::JsonWrongEntry`]). +/// +/// Each entry opened under its own selector `s` and position `j`, which its +/// ciphertext is bound to, so each is a genuine node of the row; what the +/// check adds is that it is the node the walk asked for. A subtree's rows +/// in a document are its root and then every node beneath it, at +/// consecutive positions, each node's children in order; so the rows are +/// placed one by one, each the next position after the one before, one +/// level below its parent, and carrying the selector of its parent's path +/// extended by its own label, which the key derives from the decrypted +/// label. With gates, each subtree's root is a node the path reaches: the +/// path's selector, at depth `0`, at the position the path names if its +/// last step is one. Without gates, the rows are the children of the +/// path's node and their subtrees, side by side: each child carries the +/// selector of the path extended by its label, at depth `1`, right after +/// the subtree before it. +fn place( + rows: Vec, + key: &JsonSubtreeKey, +) -> Result)>, Error> { + let below = key.gates.is_empty(); + let at = Protected::new(key.path.selector_bytes()); + let reached = key.selectors.of(at.clone())?; + let mut rest = rows.into_iter(); + let mut forest = Vec::new(); + // Without gates, the position the next child of the path's node must + // have: right after the subtree of the one before. + let mut next: Option = None; + while let Some(top) = rest.next() { + let (bytes, depth) = if below { + let bytes = child_selector_bytes(&at, &top.node.label) + .ok_or_else(|| wrong("a child of the walked node"))?; + if top.d.is_some_and(|d| d != 1) + || next.is_some_and(|j| j != top.j) + || key.selectors.of(bytes.clone())? != top.s + { + return Err(wrong("a child of the walked node")); + } + (bytes, 1) + } else { + if top.d.is_some_and(|d| d != 0) + || top.s != reached + || !reaches(&key.path, &top.node.label) + { + return Err(wrong("a node the walked path reaches")); + } + (at.clone(), 0) + }; + let mut cursor = top + .j + .checked_add(1) + .ok_or_else(|| wrong("a node of the walk"))?; + let (label, node) = take_label(top.node); + let value = rebuild_walked(node, &bytes, depth, &mut rest, &mut cursor, &key.selectors)?; + next = Some(cursor); + forest.push((label, value)); + } + Ok(forest) +} + +/// Whether a node with `label` is one `path` reaches by its last step, +/// beyond what the selector says: a position names the element at that +/// position, where the selector reads every position as the wildcard. +fn reaches(path: &JsonPath, label: &Label) -> bool { + match path.segments().last() { + Some(Segment::Index(n)) => matches!(label, Label::Index(at) if at == n), + Some(Segment::Wildcard) => matches!(label, Label::Index(_)), + Some(Segment::Key(_)) => matches!(label, Label::Key(_)), + None => matches!(label, Label::Root), + } +} + +/// The selector bytes of the child labelled `label` of the node whose +/// selector bytes are `parent`: the parent's, then the child's step, a key +/// for a member and the array step for an element, as +/// [`JsonPath::selector_bytes`] writes them. A member's key is decrypted +/// plaintext, so the bytes are built inside a `Protected`, sized before +/// they are written. `None` for the root, which is no node's child. +fn child_selector_bytes(parent: &Protected>, label: &Label) -> Option>> { + let parent_len = parent.risky_ref().len(); + let mut out = match label { + Label::Root => return None, + Label::Index(_) => Protected::new(Vec::with_capacity(parent_len + 1)), + Label::Key(key) => Protected::new(Vec::with_capacity( + parent_len + key_step_len(key.risky_ref().len()), + )), + }; + out.update_with_ref(parent.as_protected_ref(), |out, parent: &Vec| { + out.extend_from_slice(parent); + }); + match label { + Label::Key(key) => out.update_with_ref(key.as_protected_ref(), |out, key: &String| { + push_key(out, key); + }), + _ => out.update(|out| out.push(ARRAY_STEP)), + } + Some(out) +} + +/// [`rebuild`] for a walk's rows, checking each child as it is taken: +/// that it is at `cursor`, the position after the row before it; that it +/// is one level below `node`, which sits `d` levels below the walk's start; +/// and that it carries the selector of `node`'s path, whose selector bytes +/// are `bytes`, extended by its own label. A sealed root among the rows is +/// not a node of any walk. +/// +/// The rows are consecutive entries of one sealed document, placed by its +/// authenticated child counts, so they nest no deeper than the document +/// does, which is never deeper than [`MAX_DEPTH`]; a document that does is +/// refused as a shape mismatch rather than recursed into. +fn rebuild_walked( + node: Opened, + bytes: &Protected>, + d: u32, + rest: &mut impl Iterator, + cursor: &mut u64, + selectors: &Selectors, +) -> Result, Error> { + if node.whole.is_some() { + return Err(wrong("a node of the walked subtree")); + } + if d as usize > MAX_DEPTH { + return Err(shape("a JSON document no deeper than the index seals")); + } + let child = |rest: &mut dyn Iterator, cursor: &mut u64| { + let row = rest + .next() + .ok_or_else(|| shape("a complete JSON subtree"))?; + let child_bytes = child_selector_bytes(bytes, &row.node.label) + .ok_or_else(|| wrong("the next node of the walked subtree"))?; + if row.j != *cursor + || row.d.is_some_and(|got| Some(got) != d.checked_add(1)) + || selectors.of(child_bytes.clone())? != row.s + { + return Err(wrong("the next node of the walked subtree")); + } + *cursor = cursor + .checked_add(1) + .ok_or_else(|| wrong("the next node of the walked subtree"))?; + Ok::<_, Error>((row.node, child_bytes)) + }; + match node.kind { + Kind::Object => { + let mut object = protected_object(); + for _ in 0..node.children { + let (member, member_bytes) = child(rest, cursor)?; + let (label, member) = take_label(member); + let Label::Key(key) = label else { + return Err(shape("an object member")); + }; + let value = rebuild_walked(member, &member_bytes, d + 1, rest, cursor, selectors)?; + insert_member(&mut object, key, value); + } + Ok(object) + } + Kind::Array => { + let mut array = protected_array(node.children as usize); + for i in 0..node.children { + let (element, element_bytes) = child(rest, cursor)?; + if !matches!(element.label, Label::Index(at) if at == u64::from(i)) { + return Err(shape("an array element in order")); + } + push_element( + &mut array, + rebuild_walked(element, &element_bytes, d + 1, rest, cursor, selectors)?, + ); + } + Ok(array) + } + _ => node.scalar.ok_or_else(|| shape("a JSON scalar")), + } +} + /// The values of a node's children, in the node's order, moved out of it: /// an array's elements, an object's member values (each leaving `null` /// under its key, which stays in the document to be wiped with it), and @@ -1332,11 +1598,12 @@ fn open_fallback<'a, K>( /// position with the element selector, so a member that happens to sit at /// that ordinal under an object is not taken for an element. fn subtree_key_for( - prf: &HmacSha256Prf, - context: &Context<'_>, + selectors: Selectors, options: &JsonOptions, path: &JsonPath, ) -> Result { + let prf = &selectors.keyset.prf; + let context = &selectors.context; let segments = path.segments(); let (head, rest) = if options.sealed_root { let Some((first, rest)) = segments.split_first() else { @@ -1391,6 +1658,7 @@ fn subtree_key_for( token, gates, path: path.clone(), + selectors, }) } @@ -1437,7 +1705,8 @@ impl KeysetCipher<'_, K> { descriptor: NonEmpty>, ) -> Pending<'_, JsonSubtreeKey, K> { let context = descriptor.into_prf_context().into_owned(); - let key = subtree_key_for(self.prf(), &context, options, path); + let selectors = Selectors::new(Arc::clone(self.state()), context); + let key = subtree_key_for(selectors, options, path); Pending::ready(self, key.map_err(Error::from)) } } @@ -1545,10 +1814,13 @@ mod tests { #[test] fn the_subtree_key_derives_what_it_can_and_gates_the_rest() { - let context = Context::from_encoded(b"users/profile"); let value = json!({"a": {"b": [1, {"c": 2}]}, "l": [[0], 1]}); let key = |options: &JsonOptions, path: &str| { - subtree_key_for(&prf(), &context, options, &JsonPath::parse(path).unwrap()) + subtree_key_for( + Selectors::for_tests(), + options, + &JsonPath::parse(path).unwrap(), + ) }; let at = |t: &JsonTerms, path: &str| { t.nodes() @@ -1617,11 +1889,9 @@ mod tests { #[test] fn a_position_past_postgres_integer_is_refused_not_sent() { - let context = Context::from_encoded(b"users/profile"); let key = |path: &str| { subtree_key_for( - &prf(), - &context, + Selectors::for_tests(), &JsonOptions::default(), &JsonPath::parse(path).unwrap(), ) @@ -1843,6 +2113,92 @@ mod tests { assert!(matches!(sealed(longest + 1), Err(Error::Other(_)))); } + mod given_a_walked_row { + use super::*; + + fn path(text: &str) -> JsonPath { + JsonPath::parse(text).unwrap() + } + + fn key(text: &str) -> Label { + Label::Key(Protected::new(text.to_owned())) + } + + #[test] + fn a_childs_selector_bytes_are_its_paths() { + let parent = Protected::new(path("$.a[3]").selector_bytes()); + for (label, child) in [(key("b"), "$.a[*].b"), (Label::Index(7), "$.a[*][*]")] { + let bytes = child_selector_bytes(&parent, &label).expect("a child"); + assert_eq!( + bytes.risky_ref(), + &path(child).selector_bytes(), + "{child}: the parent's bytes and the child's step" + ); + assert_eq!( + bytes.risky_ref().len(), + bytes.risky_ref().capacity(), + "{child}: sized before it is written" + ); + } + assert!( + child_selector_bytes(&parent, &Label::Root).is_none(), + "the root is no node's child" + ); + } + + #[test] + fn the_last_step_of_a_path_names_the_label_it_reaches() { + assert!(reaches(&path("$.a[2]"), &Label::Index(2)), "the position"); + assert!(!reaches(&path("$.a[2]"), &Label::Index(1)), "another one"); + assert!(!reaches(&path("$.a[2]"), &key("2")), "a member"); + assert!(reaches(&path("$.a[*]"), &Label::Index(9)), "any element"); + assert!(!reaches(&path("$.a[*]"), &key("x")), "not a member"); + assert!(reaches(&path("$.a.b"), &key("b")), "a member"); + assert!(!reaches(&path("$.a.b"), &Label::Index(0)), "not an element"); + assert!(reaches(&path("$"), &Label::Root), "the root"); + assert!(!reaches(&path("$"), &key("a")), "only the root"); + } + + #[test] + fn selectors_are_the_ones_the_key_derives_and_compare_by_keyset_and_column() { + let selectors = Selectors::for_tests(); + let at = |text: &str| { + prf16( + &prf(), + SELECTOR_DOMAIN, + &Context::from_encoded(b"users/profile"), + &path(text).selector_bytes(), + ) + .unwrap() + }; + assert_eq!( + selectors + .of(Protected::new(path("$.a.b").selector_bytes())) + .unwrap(), + at("$.a.b"), + "a protected input derives what its bytes do" + ); + assert!( + selectors == Selectors::for_tests(), + "same keyset, same column" + ); + let other_column = Selectors::new( + Arc::clone(&selectors.keyset), + Context::from_encoded(b"users/bio").into_owned(), + ); + assert!(selectors != other_column, "another column"); + let other_keyset = Selectors::new( + Arc::new(KeysetState { + id: Uuid::from_u128(2), + name: None, + prf: prf(), + }), + selectors.context.clone(), + ); + assert!(selectors != other_keyset, "another keyset"); + } + } + #[test] fn too_deep_a_document_is_refused() { // The innermost scalar of `n` nested arrays sits at depth `n`. diff --git a/packages/stack-encrypt/src/json/mod.rs b/packages/stack-encrypt/src/json/mod.rs index ff59c71f0..a4ee371c0 100644 --- a/packages/stack-encrypt/src/json/mod.rs +++ b/packages/stack-encrypt/src/json/mod.rs @@ -48,8 +48,13 @@ //! `AtPath`), [`JsonLeafQuery`] (ordering at a path), and the two //! extraction keys a caller derives through the keyset cipher, //! [`JsonSelector`] and [`JsonSubtreeKey`]. What the server returns for an -//! extraction opens through [`JsonNode`] (one entry) and [`JsonSubtree`] -//! (the entries under a token). +//! extraction is [`JsonNode`] rows, which open only through what the +//! client derived for the read: [`JsonSelector::open`] for the one entry +//! `->` returns ([`JsonExtracted`]), [`JsonSubtreeKey::open`] for the rows +//! of a walk ([`JsonSubtree`]). Each checks that the rows are the ones the +//! read asked for, which their ciphertexts alone do not say: an entry is +//! sealed under its own selector and position, so any entry of the row +//! opens. //! //! # Where this deviates from the design //! @@ -528,9 +533,13 @@ where /// One entry as an extraction returns it: the entry, the document's header /// grafted beside it, and the entry's ordinal `j` in the document, which -/// its ciphertext is bound to. What `->` and the walk return per row, and -/// what [`DecryptInto`](crate::DecryptInto) opens to the node's -/// scalar, or to the whole document for a sealed root. +/// its ciphertext is bound to. What `->` and the walk return per row. +/// +/// A row is the database's word, so it does not open on its own: the +/// selector the client derived opens the entry `->` returned +/// ([`JsonSelector::open`]), and the key that keyed a walk opens its rows +/// ([`JsonSubtreeKey::open`]), each after checking the rows are the ones +/// it asked for. /// /// Deserializes from the entry's wire fields plus `h` and `j`, and tolerates /// the envelope fields the SQL grafts beside them (`i`, `v`, `d`). @@ -562,10 +571,43 @@ impl JsonNode { } } -impl Decryptable for JsonNode { +/// The entry `->` returned, with the selector the client asked for: what +/// [`DecryptInto`](crate::DecryptInto) opens to the node's scalar, +/// or to the whole document for a sealed root asked for at `$`. Built only +/// by [`JsonSelector::open`], so it always carries the selector the entry +/// is checked against; an entry carrying another is refused with +/// [`Error::JsonWrongEntry`](crate::Error::JsonWrongEntry) before any key +/// is retrieved. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct JsonExtracted { + node: JsonNode, + selector: Tag, +} + +impl JsonExtracted { + pub(crate) fn keyed(node: JsonNode, selector: Tag) -> Self { + Self { node, selector } + } + + /// The row, as the database returned it. + pub fn node(&self) -> &JsonNode { + &self.node + } + + /// The selector the read asked for. + pub fn selector(&self) -> JsonSelector { + JsonSelector(self.selector) + } + + pub(crate) fn into_parts(self) -> (JsonNode, Tag) { + (self.node, self.selector) + } +} + +impl Decryptable for JsonExtracted { const DECRYPTABLE: bool = true; } -impl DecryptField for JsonNode +impl DecryptField for JsonExtracted where Self: crate::DecryptInto

, Ctx: Into<>::Context>, @@ -580,7 +622,9 @@ where /// token, gates)` yields, collected, and what /// [`DecryptInto`](crate::DecryptInto) reassembles into the value /// under the walked path. Built only by the key that keyed the walk -/// ([`JsonSubtreeKey::open`]), so it always carries the walked path. +/// ([`JsonSubtreeKey::open`]), so it always carries the walked path, and +/// what checks every row against it when the rows are opened (see +/// [`JsonSubtreeKey::open`] for the checks). /// /// The ungated walk returns the node's descendants and not the node /// itself, so the node's kind is read from its children's labels: elements @@ -599,15 +643,15 @@ where #[derive(Clone, Debug, PartialEq, Eq)] pub struct JsonSubtree { nodes: Vec, - path: JsonPath, - below: bool, + key: JsonSubtreeKey, } impl JsonSubtree { - /// The rows of a walk keyed from `path`: below the path's node when - /// `below` (`json_tree_subtree`), at the nodes it reaches otherwise. - pub(crate) fn keyed(nodes: Vec, path: JsonPath, below: bool) -> Self { - Self { nodes, path, below } + /// The rows of a walk `key` keyed: below the path's node when the key + /// has no gates (`json_tree_subtree`), at the nodes it reaches + /// otherwise. + pub(crate) fn keyed(nodes: Vec, key: JsonSubtreeKey) -> Self { + Self { nodes, key } } /// The rows. @@ -617,7 +661,18 @@ impl JsonSubtree { /// The path the walk was keyed from. pub fn path(&self) -> &JsonPath { - &self.path + &self.key.path + } + + /// Whether the rows are the descendants of the path's node + /// (`json_tree_subtree`, a key with no gates) rather than the nodes + /// the path reaches. + pub(crate) fn below(&self) -> bool { + self.key.gates.is_empty() + } + + pub(crate) fn into_parts(self) -> (Vec, JsonSubtreeKey) { + (self.nodes, self.key) } /// Whether the rows are the sealed root the server returned in place diff --git a/packages/stack-encrypt/src/json/path.rs b/packages/stack-encrypt/src/json/path.rs index 2545defe1..77f86a457 100644 --- a/packages/stack-encrypt/src/json/path.rs +++ b/packages/stack-encrypt/src/json/path.rs @@ -62,7 +62,7 @@ impl crate::ErrorPayload for JsonPathError { const KEY_STEP: u8 = 0x01; /// The tag byte of an array step (positional or wildcard) in a path's /// canonical bytes. -const ARRAY_STEP: u8 = 0x02; +pub(crate) const ARRAY_STEP: u8 = 0x02; impl JsonPath { /// The root, `$`. @@ -308,13 +308,21 @@ impl JsonPath { } } -/// A key step: the tag, the key's length and the key. -fn push_key(out: &mut Vec, key: &str) { +/// A key step: the tag, the key's length and the key. What +/// [`JsonPath::selector_bytes`] writes for a key, and what a walk's rows +/// are checked against when the client extends a path by a member's +/// decrypted key. +pub(crate) fn push_key(out: &mut Vec, key: &str) { out.push(KEY_STEP); out.extend_from_slice(&(key.len() as u32).to_le_bytes()); out.extend_from_slice(key.as_bytes()); } +/// How many bytes [`push_key`] writes for a key of `len` bytes. +pub(crate) const fn key_step_len(len: usize) -> usize { + 1 + 4 + len +} + /// The JSON string at the front of `text`, which starts at its opening /// quote: its value and the text after its closing quote. The escapes are /// JSON's. The refusal is the reason; the caller names the offset. diff --git a/packages/stack-encrypt/src/json/query.rs b/packages/stack-encrypt/src/json/query.rs index e515723ce..1600fbe47 100644 --- a/packages/stack-encrypt/src/json/query.rs +++ b/packages/stack-encrypt/src/json/query.rs @@ -12,9 +12,9 @@ use std::fmt; use serde::{Deserialize, Serialize}; use serde_json::Value; -use super::derive::{JsonTerms, TermNode}; +use super::derive::{JsonTerms, Selectors, TermNode}; use super::path::JsonPath; -use super::{base64, JsonDocument, JsonNode, JsonOptions, JsonSubtree, Tag}; +use super::{base64, JsonDocument, JsonExtracted, JsonNode, JsonOptions, JsonSubtree, Tag}; use crate::op::{ Answers, AtPath, Between, ContainedBy, Contains, DefaultIndex, Eq, Gt, Gte, Input, LabelledTarget, Lt, Lte, One, Pair, Pathed, QueryTerm, QueryToken, Supports, @@ -296,6 +296,10 @@ impl<'de> Deserialize<'de> for JsonLeafQuery { /// A path selector: what `->` and the path functions take. Prints and /// serializes as the base64 the stored entry carries in `s`. +/// +/// It is also what opens the entry `->` returns ([`open`](Self::open)): +/// the selector the client derived for the path is the one the entry must +/// carry, so a database that answers with another path's entry is refused. #[derive(Clone, Copy, PartialEq, Eq, Hash)] pub struct JsonSelector(pub(crate) Tag); @@ -304,6 +308,24 @@ impl JsonSelector { pub fn as_bytes(&self) -> &Tag { &self.0 } + + /// The entry `col -> selector` returned, ready to open: the only way to + /// open one. Opening it ([`DecryptInto`](crate::DecryptInto)) + /// first checks that the entry carries this selector, and refuses any + /// other with [`Error::JsonWrongEntry`] before a key is retrieved: an + /// entry's ciphertext is bound to the selector it carries, so a + /// database that answers `$.salary` with the row's `$.bonus`, or with + /// the sealed root (the whole document, whose selector is the root's), + /// returns an entry that would open, and is not the one asked for. + /// + /// What the check covers is the path, read as a selector reads it: a + /// position reads as the wildcard, so any element at the path passes, + /// as `->` returns the first one. It does not cover the row: the same + /// path's entry from another row of the column, returned with that + /// row's header, opens (ADR 0010). + pub fn open(&self, node: JsonNode) -> JsonExtracted { + JsonExtracted::keyed(node, self.0) + } } impl fmt::Display for JsonSelector { @@ -355,11 +377,18 @@ impl Gate { /// derived from, which [`open`](Self::open) attaches to the rows the walk /// returns. Serializes as `{"t": "…", "g": ["…", "#0:…", …]}`, `g` omitted /// when empty; the path is not part of the wire. +/// +/// The key also holds what derives selectors under the column it was +/// derived for (the keyset's index-key PRF, shared with the keyset handle, +/// and the column's context), so that opening the rows can check each +/// one's selector against the path it must sit at. Neither is printed or +/// serialized. #[derive(Clone, PartialEq, Eq)] pub struct JsonSubtreeKey { pub(crate) token: Tag, pub(crate) gates: Vec, pub(crate) path: JsonPath, + pub(crate) selectors: Selectors, } impl JsonSubtreeKey { @@ -397,8 +426,25 @@ impl JsonSubtreeKey { /// token, gates)` and are the nodes the path reaches, with theirs. The /// key knows which, so the fallback opens to what the walked rows /// open to either way. + /// + /// Opening the rows checks each one against the key, and refuses the + /// rows with [`Error::JsonWrongEntry`] if one does not fit: a node the + /// path reaches must carry the path's selector, at depth `0`, with the + /// position the path names if it ends in one; every other row must + /// carry the selector of its parent's path extended by its own label + /// (its key, or the wildcard for an array element), sit one level + /// below its parent, and follow the row before it in the document with + /// no entry between, so that each node's rows are exactly its subtree. + /// A row whose parent is not among the rows is refused. + /// + /// Not covered: which row of the column the entries come from (ADR + /// 0010), and, until the walk returns the node it starts from, which + /// node at the path's selector a walk with no gates descends from. An + /// array position before the last step of a path reads as the wildcard + /// to the selectors, so another element's subtree at the same depth + /// passes too. pub fn open(&self, rows: Vec) -> JsonSubtree { - JsonSubtree::keyed(rows, self.path.clone(), self.gates.is_empty()) + JsonSubtree::keyed(rows, self.clone()) } } @@ -782,6 +828,7 @@ mod tests { token: [0xcd; 16], gates: vec![Gate::Selector([0xab; 16]), Gate::Slot(3, [0xab; 16])], path: JsonPath::parse("$.a.b[3]").unwrap(), + selectors: Selectors::for_tests(), }; assert_eq!(key.token(), "zc3Nzc3Nzc3Nzc3Nzc3NzQ=="); assert_eq!(key.token_bytes(), &[0xcd; 16]); @@ -802,6 +849,7 @@ mod tests { token: [0xcd; 16], gates: Vec::new(), path: JsonPath::parse("$.a").unwrap(), + selectors: Selectors::for_tests(), }; assert_eq!( serde_json::to_value(&bare).unwrap(), diff --git a/packages/stack-encrypt/src/keyset.rs b/packages/stack-encrypt/src/keyset.rs index 78638ddc5..c1d891e5a 100644 --- a/packages/stack-encrypt/src/keyset.rs +++ b/packages/stack-encrypt/src/keyset.rs @@ -614,6 +614,13 @@ impl<'k, K> KeysetCipher<'k, K> { &self.state.prf } + /// The keyset's loaded state, shared: for a value that must derive + /// terms after this handle is gone, such as a JSON subtree key, which + /// derives the selectors a walk's rows must carry when it opens them. + pub(crate) fn state(&self) -> &Arc { + &self.state + } + /// The underlying data-key source. pub fn kms(&self) -> &'k K { self.cipher.kms() diff --git a/packages/stack-encrypt/tests/json_index.rs b/packages/stack-encrypt/tests/json_index.rs index 98ec5a831..420e82254 100644 --- a/packages/stack-encrypt/tests/json_index.rs +++ b/packages/stack-encrypt/tests/json_index.rs @@ -12,7 +12,8 @@ mod common; use common::stack_cipher; use serde_json::{json, Map, Value}; use stack_encrypt::json::{ - JsonDocument, JsonNode, JsonOptions, JsonPath, JsonQuery, JsonSubtree, JsonSubtreeKey, + JsonDocument, JsonExtracted, JsonNode, JsonOptions, JsonPath, JsonQuery, JsonSelector, + JsonSubtree, JsonSubtreeKey, }; use stack_encrypt::op; use stack_encrypt::sem::TermError; @@ -309,6 +310,28 @@ async fn subtree_key( .await } +/// The selector of `path` under the suite's column. +async fn selector_of(cipher: &StackCipher, path: &str) -> JsonSelector { + cipher + .default_keyset() + .json_selector( + &JsonPath::parse(path).unwrap(), + nonempty!("users").with("profile"), + ) + .await + .unwrap() +} + +/// `node` as `->` on `path` returned it, opened by the selector derived +/// for `path`: the only way to open an extracted entry. +async fn extracted( + cipher: &StackCipher, + path: &str, + node: JsonNode, +) -> JsonExtracted { + selector_of(cipher, path).await.open(node) +} + /// `rows` as the key derived for `path` opens them: the only way to build /// a [`JsonSubtree`]. async fn keyed( @@ -551,16 +574,24 @@ async fn an_extracted_node_opens_to_its_scalar_and_a_subtree_to_its_value() { let cipher = stack_cipher().await; let doc = seal(&cipher, &fixture()).await; let node = |j: usize| doc.node(j).unwrap(); - let scalar = |j: usize| cipher.decrypt_as::(node(j), aead()); - assert_eq!(scalar(1).await.unwrap(), json!("world-1")); - assert_eq!(scalar(6).await.unwrap(), json!(1)); + let scalar = |path: &'static str, j: usize| { + let cipher = &cipher; + let node = node(j); + async move { + cipher + .decrypt_as::(extracted(cipher, path, node).await, aead()) + .await + } + }; + assert_eq!(scalar("$.hello", 1).await.unwrap(), json!("world-1")); + assert_eq!(scalar("$.nested.list[0].x", 6).await.unwrap(), json!(1)); assert_eq!( - scalar(0).await.unwrap(), + scalar("$", 0).await.unwrap(), fixture(), "the sealed root is the document" ); assert!( - matches!(scalar(2).await, Err(Error::OpenedType { .. })), + matches!(scalar("$.nested", 2).await, Err(Error::OpenedType { .. })), "a container node holds no value of its own" ); @@ -583,20 +614,14 @@ async fn an_extracted_node_opens_to_its_scalar_and_a_subtree_to_its_value() { .unwrap(); assert_eq!(forest, vec![json!("constant"), json!([{"x": 1}, {"x": 2}])]); - // The list's token is the server's to derive, one level under nested. - let list = doc - .children(key.token_bytes()) - .into_iter() - .find(|(j, _, _)| *j == 4) - .map(|(_, _, t)| t) + // The list is one gate below nested: the walk reaches it and returns + // it with its subtree. + let list = subtree_key(&cipher, "$.nested.list", JsonOptions::default()) + .await .unwrap(); - let rows: Vec = doc - .subtree(&list) - .iter() - .map(|(j, _, _)| node(*j)) - .collect(); + let rows = gated_walk(&doc, &list); let elements = cipher - .decrypt_as::(keyed(&cipher, "$.nested.list", rows).await, aead()) + .decrypt_as::(list.open(rows), aead()) .await .unwrap(); assert_eq!(elements, json!([{"x": 1}, {"x": 2}])); @@ -610,7 +635,10 @@ async fn an_extracted_node_opens_to_its_scalar_and_a_subtree_to_its_value() { assert_eq!(back.ordinal(), 6); assert_eq!( cipher - .decrypt_as::(back, aead()) + .decrypt_as::( + extracted(&cipher, "$.nested.list[*].x", back).await, + aead() + ) .await .unwrap(), json!(1) @@ -620,8 +648,8 @@ async fn an_extracted_node_opens_to_its_scalar_and_a_subtree_to_its_value() { /// The rows of a walk come from the server, which can repeat an entry or /// pick which entries to return. Every entry is authentic, but the rebuild /// nests the containers it is handed: a repeated row is refused, and so is -/// a line of containers deeper than any document the index seals, as a -/// shape mismatch rather than a recursion as deep as the rows. +/// a line of containers that are not one subtree's consecutive entries, +/// before the rebuild can recurse as deep as the rows. #[tokio::test] async fn subtree_rows_the_server_repeats_or_lines_up_are_refused() { let cipher = stack_cipher().await; @@ -657,7 +685,7 @@ async fn subtree_rows_the_server_repeats_or_lines_up_are_refused() { aead() ) .await, - Err(Error::OpenedType { .. }) + Err(Error::JsonWrongEntry { .. }) )); // The same through arrays: one outer array, then every key's inner @@ -678,7 +706,7 @@ async fn subtree_rows_the_server_repeats_or_lines_up_are_refused() { cipher .decrypt_as::, JsonSubtree>(keyed(&cipher, "$.k000", arrays).await, aead()) .await, - Err(Error::OpenedType { .. }) + Err(Error::JsonWrongEntry { .. }) )); } @@ -745,7 +773,9 @@ async fn a_moved_reordered_or_regrafted_entry_fails_to_open() { lifted["j"] = json!(2); let lifted: JsonNode = serde_json::from_value(lifted).unwrap(); assert!(matches!( - cipher.decrypt_as::(lifted, aead()).await, + cipher + .decrypt_as::(extracted(&cipher, "$.a", lifted).await, aead()) + .await, Err(Error::Aead) )); } @@ -1548,18 +1578,21 @@ async fn gated_walk_rows_open_to_the_reached_nodes_value() { } other => panic!("{other:?}"), } - // Without a d = 0 row the same entries are the children of one node and - // assemble into it. + // The key says which walk the rows came from: the children of the + // list, which an ungated walk from the list's token would return, are + // not what the gated walk to the list returns, and are refused. let rows: Vec = doc .subtree(&list) .iter() .map(|(j, _, _)| doc.node(*j).unwrap()) .collect(); - let elements = cipher + let refused = cipher .decrypt_as::(keyed(&cipher, "$.nested.list", rows).await, aead()) - .await - .unwrap(); - assert_eq!(elements, json!([{"x": 1}, {"x": 2}])); + .await; + assert!( + matches!(refused, Err(Error::JsonWrongEntry { .. })), + "{refused:?}" + ); } /// A document, an extracted node and a walked subtree are each the one @@ -1569,24 +1602,18 @@ async fn gated_walk_rows_open_to_the_reached_nodes_value() { async fn each_stored_form_opens_as_a_record_field() { let cipher = stack_cipher().await; let doc = seal(&cipher, &fixture()).await; - let node = doc.node(1).unwrap(); - let key = subtree_key(&cipher, "$.nested", JsonOptions::default()) + let node = extracted(&cipher, "$.hello", doc.node(1).unwrap()).await; + let list = subtree_key(&cipher, "$.nested.list", JsonOptions::default()) .await .unwrap(); - let list = doc - .children(key.token_bytes()) - .into_iter() - .find(|(j, _, _)| *j == 4) - .map(|(_, _, t)| t) - .unwrap(); - let rows: Vec = doc - .subtree(&list) - .iter() - .map(|(j, _, _)| doc.node(*j).unwrap()) - .collect(); - let subtree = keyed(&cipher, "$.nested.list", rows.clone()).await; + let rows = gated_walk(&doc, &list); + let subtree = list.open(rows.clone()); assert_eq!(subtree.nodes(), rows.as_slice()); - assert_eq!(subtree.nodes().len(), 4, "both elements and both leaves"); + assert_eq!( + subtree.nodes().len(), + 5, + "the list, both elements and both leaves" + ); let opening: Option> = doc.decryption_field(aead()); let opened = cipher @@ -1669,14 +1696,18 @@ async fn the_sealed_root_a_walk_fell_back_to_opens_at_the_path() { } // The same root at depth 1, or at depth 0 under an unsealed document, - // is a walk's row and not the fallback; neither assembles. + // is a walk's row and not the fallback, and not a child of the walked + // node either: its selector is the root's. let mut as_walked = serde_json::to_value(doc.node(0).unwrap()).unwrap(); as_walked["d"] = json!(1); let as_walked: JsonNode = serde_json::from_value(as_walked).unwrap(); let rows = keyed(&cipher, "$.nested", vec![as_walked]).await; assert!(!rows.is_sealed_root()); let refused = cipher.decrypt_as::(rows, aead()).await; - assert!(matches!(refused, Err(Error::OpenedType { .. }))); + assert!( + matches!(refused, Err(Error::JsonWrongEntry { .. })), + "{refused:?}" + ); let open_doc = seal_with(&cipher, &fixture(), &open_root()).await; assert!(!keyed(&cipher, "$.nested", vec![fallback(&open_doc)]) .await diff --git a/packages/stack-encrypt/tests/json_read_checks.rs b/packages/stack-encrypt/tests/json_read_checks.rs new file mode 100644 index 000000000..963b37ac8 --- /dev/null +++ b/packages/stack-encrypt/tests/json_read_checks.rs @@ -0,0 +1,531 @@ +//! What the client checks when it opens the rows a JSON index read +//! returned, with no database: the entry `->` returns must carry the +//! selector the client derived for the path, and every row of a subtree +//! walk must sit where the key that keyed the walk says its rows are. +//! Each entry's ciphertext is bound to its own selector and position, so +//! without these checks any entry of the row would open in place of the one +//! asked for. +//! +//! The rows here are built as the SQL returns them (the entry, the header +//! and the ordinal grafted, and a walk's depth), from a sealed document, so +//! every entry in them is genuine. What varies is which entries a dishonest +//! database hands back. + +mod common; + +use std::sync::atomic::Ordering; + +use common::CountingSource; +use serde_json::{json, Value}; +use stack_encrypt::json::{ + JsonDocument, JsonExtracted, JsonNode, JsonOptions, JsonPath, JsonSelector, JsonSubtree, + JsonSubtreeKey, +}; +use stack_encrypt::target::{AeadContext, CallerContext}; +use stack_encrypt::{nonempty, Error, StackCipher}; + +type Cipher = StackCipher; + +async fn cipher() -> (Cipher, std::sync::Arc) { + let source = CountingSource::new(); + let (_, retrieved) = source.counters(); + let cipher = StackCipher::builder() + .kms(source) + .init() + .await + .expect("build cipher"); + (cipher, retrieved) +} + +fn aead() -> AeadContext { + AeadContext::from(nonempty!("users").with("profile")) +} + +/// `$`, `hello` (1), `nested` (2), `deep` (3), `list` (4), its elements (5, +/// 7) and their `x` (6, 8), `number` (9): depth-first, members in key order. +fn fixture() -> Value { + json!({"hello": "world-1", "nested": {"deep": "constant", "list": [{"x": 1}, {"x": 2}]}, "number": 1}) +} + +async fn seal(cipher: &Cipher, value: &Value) -> JsonDocument { + cipher + .default_keyset() + .encrypt_as::( + value, + CallerContext::from(nonempty!("users").with("profile")), + ) + .await + .expect("seals") +} + +async fn selector(cipher: &Cipher, path: &str) -> JsonSelector { + cipher + .default_keyset() + .json_selector( + &JsonPath::parse(path).unwrap(), + nonempty!("users").with("profile"), + ) + .await + .unwrap() +} + +async fn key(cipher: &Cipher, path: &str) -> JsonSubtreeKey { + key_with(cipher, path, &JsonOptions::default()).await +} + +async fn key_with(cipher: &Cipher, path: &str, options: &JsonOptions) -> JsonSubtreeKey { + cipher + .default_keyset() + .json_subtree_key( + &JsonPath::parse(path).unwrap(), + options, + nonempty!("users").with("profile"), + ) + .await + .unwrap() +} + +async fn seal_with(cipher: &Cipher, value: &Value, options: &JsonOptions) -> JsonDocument { + cipher + .default_keyset() + .run( + stack_encrypt::target::json_document::<_, stack_encrypt::target::Borrowed>( + options.clone(), + ), + value, + CallerContext::from(nonempty!("users").with("profile")), + ) + .await + .expect("seals") +} + +/// The options with the root not sealed. +fn open_root() -> JsonOptions { + JsonOptions { + sealed_root: false, + ..JsonOptions::new() + } +} + +/// The entry at `j` as an extractor returns it, at depth `d` when a walk +/// returned it. +fn row(doc: &JsonDocument, j: usize, d: Option) -> JsonNode { + let mut wire = serde_json::to_value(doc.node(j).unwrap()).unwrap(); + if let Some(d) = d { + wire["d"] = json!(d); + } + serde_json::from_value(wire).unwrap() +} + +/// The rows the SQL walk returns for `key`, walked as it walks: with no +/// gates, every descendant of the token's node from depth 1; with gates, +/// through each gate (a selector, or `#n:`), then each node the +/// gates reach at depth 0 with everything beneath it. +fn walk(doc: &JsonDocument, key: &JsonSubtreeKey) -> Vec { + let gates = key.gates(); + if gates.is_empty() { + return doc + .subtree(key.token_bytes()) + .iter() + .map(|(j, _, d)| row(doc, *j, Some(*d))) + .collect(); + } + let mut frontier = vec![(None, *key.token_bytes())]; + for gate in &gates { + let mut next = Vec::new(); + for (_, token) in &frontier { + let children = doc.children(token); + match gate.strip_prefix('#') { + Some(slot) => { + let (n, wanted) = slot.split_once(':').unwrap(); + if let Some((j, entry, t)) = children.get(n.parse::().unwrap()) { + if base64(entry.selector()) == wanted { + next.push((Some(*j), *t)); + } + } + } + None => next.extend( + children + .iter() + .filter(|(_, entry, _)| base64(entry.selector()) == *gate) + .map(|(j, _, t)| (Some(*j), *t)), + ), + } + } + frontier = next; + } + let mut rows = Vec::new(); + for (j, token) in frontier { + rows.push(row(doc, j.unwrap(), Some(0))); + rows.extend( + doc.subtree(&token) + .iter() + .map(|(j, _, d)| row(doc, *j, Some(*d))), + ); + } + rows +} + +fn base64(bytes: &[u8]) -> String { + use base64ct::{Base64, Encoding}; + Base64::encode_string(bytes) +} + +async fn open_extracted(cipher: &Cipher, extracted: JsonExtracted) -> Result { + cipher + .decrypt_as::(extracted, aead()) + .await +} + +async fn open_value(cipher: &Cipher, rows: JsonSubtree) -> Result { + cipher.decrypt_as::(rows, aead()).await +} + +async fn open_list(cipher: &Cipher, rows: JsonSubtree) -> Result, Error> { + cipher + .decrypt_as::, JsonSubtree>(rows, aead()) + .await +} + +mod assertions { + use super::*; + + /// The read was refused as the wrong entry, naming `expected`. + pub(crate) fn wrong_entry(result: Result, expected: &str) { + match result { + Err(Error::JsonWrongEntry { expected: got }) => { + assert_eq!(got, expected, "the refusal names what the read expected") + } + other => panic!("expected JsonWrongEntry ({expected}), got {other:?}"), + } + } +} + +mod given_the_entry_arrow_returned { + use super::*; + + #[tokio::test] + async fn the_entry_at_the_path_opens_to_its_scalar() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let hello = selector(&cipher, "$.hello").await; + let opened = open_extracted(&cipher, hello.open(row(&doc, 1, None))).await; + assert_eq!(opened.unwrap(), json!("world-1"), "the value at the path"); + } + + #[tokio::test] + async fn another_paths_entry_is_refused_before_any_key_is_retrieved() { + let (cipher, retrieved) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let hello = selector(&cipher, "$.hello").await; + // `$.number` is a genuine entry of the row, and opens on its own + // selector: the database answered `$.hello` with it. + let extracted = hello.open(row(&doc, 9, None)); + assert_eq!(extracted.selector(), hello, "the selector asked for"); + assert_eq!(extracted.node().ordinal(), 9, "the row as returned"); + assert_wrong(open_extracted(&cipher, extracted).await); + assert_eq!( + retrieved.load(Ordering::SeqCst), + 0, + "the refusal retrieved no key" + ); + } + + #[tokio::test] + async fn the_sealed_root_is_refused_in_place_of_a_node() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let hello = selector(&cipher, "$.hello").await; + // The sealed root opens to the whole document: every field, for a + // read of one. + assert_wrong(open_extracted(&cipher, hello.open(row(&doc, 0, None))).await); + } + + #[tokio::test] + async fn the_sealed_root_opens_when_the_root_was_asked_for() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let root = selector(&cipher, "$").await; + let opened = open_extracted(&cipher, root.open(row(&doc, 0, None))).await; + assert_eq!(opened.unwrap(), fixture(), "`$` is the whole document"); + } + + #[tokio::test] + async fn an_entry_relabelled_with_the_selector_asked_for_fails_to_open() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let hello = selector(&cipher, "$.hello").await; + // Passing the check by rewriting `s` breaks the ciphertext's + // binding to its own selector instead. + let mut wire = serde_json::to_value(doc.node(9).unwrap()).unwrap(); + wire["s"] = json!(hello.to_string()); + let relabelled: JsonNode = serde_json::from_value(wire).unwrap(); + let opened = open_extracted(&cipher, hello.open(relabelled)).await; + assert!(matches!(opened, Err(Error::Aead)), "{opened:?}"); + } + + #[tokio::test] + async fn any_element_at_the_path_opens_since_a_selector_reads_a_position_as_the_wildcard() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let second = selector(&cipher, "$.nested.list[1].x").await; + // `->` returns the first element at the path; the selector cannot + // say which position was meant. + let opened = open_extracted(&cipher, second.open(row(&doc, 6, None))).await; + assert_eq!(opened.unwrap(), json!(1), "the first element's x"); + } + + #[tokio::test] + async fn the_same_path_from_another_row_opens_by_design() { + let (cipher, _) = cipher().await; + let other = seal(&cipher, &json!({"hello": "world-2"})).await; + let hello = selector(&cipher, "$.hello").await; + // Row provenance is the database's word (ADR 0010): an entry + // returned with its own row's header opens. + let opened = open_extracted(&cipher, hello.open(row(&other, 1, None))).await; + assert_eq!(opened.unwrap(), json!("world-2"), "another row's value"); + } + + fn assert_wrong(result: Result) { + assertions::wrong_entry(result, "the entry at the path asked for"); + } +} + +mod given_the_rows_of_a_walk_without_gates { + use super::*; + + const CHILD: &str = "a child of the walked node"; + const NEXT: &str = "the next node of the walked subtree"; + + #[tokio::test] + async fn the_honest_rows_open_to_the_nodes_value() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let nested = key(&cipher, "$.nested").await; + assert!(nested.gates().is_empty(), "a chain head's token"); + let rows = walk(&doc, &nested); + assert_eq!(rows.len(), 6, "nested's descendants"); + let value = open_value(&cipher, nested.open(rows.clone())).await; + assert_eq!(value.unwrap(), fixture()["nested"], "the sub-document"); + let list = open_list(&cipher, nested.open(rows)).await; + assert_eq!( + list.unwrap(), + vec![json!("constant"), json!([{"x": 1}, {"x": 2}])], + "the members' values" + ); + } + + #[tokio::test] + async fn the_rows_of_another_field_of_the_same_shape_are_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &json!({"a": {"x": 1}, "b": {"x": 2}})).await; + let a = key(&cipher, "$.a").await; + let b = key(&cipher, "$.b").await; + // The database answers `$.a` with `$.b`'s subtree: one genuine + // member, in its place, but under `$.b`'s path. + let refused = open_value(&cipher, a.open(walk(&doc, &b))).await; + assertions::wrong_entry(refused, CHILD); + } + + #[tokio::test] + async fn a_row_of_another_path_spliced_in_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let nested = key(&cipher, "$.nested").await; + // `number` (9) sits right after nested's subtree in the document, + // so it follows the rows without a gap; its selector is not a + // member of nested's. + let mut rows = walk(&doc, &nested); + rows.push(row(&doc, 9, Some(1))); + let refused = open_value(&cipher, nested.open(rows)).await; + assertions::wrong_entry(refused, CHILD); + } + + #[tokio::test] + async fn a_dropped_row_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let nested = key(&cipher, "$.nested").await; + // The first element (5) is dropped, so its `x` (6) would be taken + // for the element itself. + let rows: Vec = walk(&doc, &nested) + .into_iter() + .filter(|n| n.ordinal() != 5) + .collect(); + let refused = open_value(&cipher, nested.open(rows)).await; + assertions::wrong_entry(refused, NEXT); + } + + #[tokio::test] + async fn a_row_reported_at_the_wrong_depth_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let nested = key(&cipher, "$.nested").await; + let rows: Vec = walk(&doc, &nested) + .into_iter() + .map(|n| { + if n.ordinal() == 6 { + row(&doc, 6, Some(1)) + } else { + n + } + }) + .collect(); + let refused = open_value(&cipher, nested.open(rows)).await; + assertions::wrong_entry(refused, NEXT); + } + + #[tokio::test] + async fn a_member_reported_at_the_wrong_depth_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let nested = key(&cipher, "$.nested").await; + let rows: Vec = walk(&doc, &nested) + .into_iter() + .map(|n| { + if n.ordinal() == 3 { + row(&doc, 3, Some(2)) + } else { + n + } + }) + .collect(); + let refused = open_value(&cipher, nested.open(rows)).await; + assertions::wrong_entry(refused, CHILD); + } + + #[tokio::test] + async fn an_orphan_whose_parent_was_left_out_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let nested = key(&cipher, "$.nested").await; + // `deep` (3) and then the second element's `x` (8) alone: its + // parent and everything between are missing. + let rows = vec![row(&doc, 3, Some(1)), row(&doc, 8, Some(3))]; + let refused = open_value(&cipher, nested.open(rows)).await; + assertions::wrong_entry(refused, CHILD); + } + + #[tokio::test] + async fn a_gap_between_two_members_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &json!({"a": {"x": 1, "y": {"z": 2}, "w": 3}})).await; + let a = key(&cipher, "$.a").await; + // `$.a`'s members are w (2), x (3) and y (4) with z (5) below: the + // database leaves x out, and w and y are each genuine members. + let rows: Vec = walk(&doc, &a) + .into_iter() + .filter(|n| n.ordinal() != 3) + .collect(); + let refused = open_value(&cipher, a.open(rows)).await; + assertions::wrong_entry(refused, CHILD); + } + + #[tokio::test] + async fn the_deepest_document_the_index_seals_opens_from_the_open_root() { + let (cipher, _) = cipher().await; + // 127 arrays: the innermost scalar 127 levels below the root, the + // deepest the index seals. + let mut value = json!(1); + for _ in 0..127 { + value = json!([value]); + } + let doc = seal_with(&cipher, &value, &open_root()).await; + let root = key_with(&cipher, "$", &open_root()).await; + let rows = walk(&doc, &root); + assert_eq!(rows.len(), 127, "every node below the root"); + let opened = open_value(&cipher, root.open(rows)).await; + assert_eq!(opened.unwrap(), value, "the whole document"); + } + + #[tokio::test] + async fn rows_with_no_reported_depth_are_placed_by_the_document() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let nested = key(&cipher, "$.nested").await; + // The depth is the database's report; the rows' positions and the + // opened nodes place them, so rows without it open the same. + let rows: Vec = walk(&doc, &nested) + .iter() + .map(|n| row(&doc, n.ordinal() as usize, None)) + .collect(); + let value = open_value(&cipher, nested.open(rows)).await; + assert_eq!(value.unwrap(), fixture()["nested"], "the sub-document"); + } +} + +mod given_the_rows_of_a_gated_walk { + use super::*; + + const REACHED: &str = "a node the walked path reaches"; + + #[tokio::test] + async fn the_honest_rows_open_to_the_reached_nodes() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let second = key(&cipher, "$.nested.list[1]").await; + let value = open_value(&cipher, second.open(walk(&doc, &second))).await; + assert_eq!(value.unwrap(), json!({"x": 2}), "the second element"); + let xs = key(&cipher, "$.nested.list[*].x").await; + let values = open_list(&cipher, xs.open(walk(&doc, &xs))).await; + assert_eq!(values.unwrap(), vec![json!(1), json!(2)], "every x"); + } + + #[tokio::test] + async fn another_position_of_the_array_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let second = key(&cipher, "$.nested.list[1]").await; + // The first element carries the same selector as the second: the + // position in its label tells them apart. + let first = key(&cipher, "$.nested.list[0]").await; + let refused = open_value(&cipher, second.open(walk(&doc, &first))).await; + assertions::wrong_entry(refused, REACHED); + } + + #[tokio::test] + async fn a_node_of_another_path_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let xs = key(&cipher, "$.nested.list[*].x").await; + let refused = open_list(&cipher, xs.open(vec![row(&doc, 3, Some(0))])).await; + assertions::wrong_entry(refused, REACHED); + } + + #[tokio::test] + async fn a_descendant_left_without_its_reached_node_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let list = key(&cipher, "$.nested.list").await; + // The list's own row is dropped: what is left are its children, + // which the gated walk does not return on their own. + let rows: Vec = walk(&doc, &list) + .into_iter() + .filter(|n| n.ordinal() != 4) + .collect(); + let refused = open_value(&cipher, list.open(rows)).await; + assertions::wrong_entry(refused, REACHED); + } + + #[tokio::test] + async fn a_reached_node_reported_below_depth_zero_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let xs = key(&cipher, "$.nested.list[*].x").await; + let refused = open_list(&cipher, xs.open(vec![row(&doc, 6, Some(1))])).await; + assertions::wrong_entry(refused, REACHED); + } + + #[tokio::test] + async fn the_sealed_root_among_walked_rows_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let list = key(&cipher, "$.nested.list").await; + // The root at depth 1 is not the fallback, and its selector is not + // the list's. + let mut rows = walk(&doc, &list); + rows.push(row(&doc, 0, Some(1))); + let refused = open_value(&cipher, list.open(rows)).await; + assertions::wrong_entry(refused, REACHED); + } +} From c3d7fee743ef4ef2e656a2267aaf297d767891bc Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 24 Oct 2026 19:07:08 +1100 Subject: [PATCH 2/5] docs(plans): name the pull request that took the extracted-entry check The design record marks the Phase 4 bullet on extracted entries taken; it now names the pull request that took it, #1358. Refs #1355 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- docs/plans/2026-10-17-json-tree-index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/plans/2026-10-17-json-tree-index.md b/docs/plans/2026-10-17-json-tree-index.md index 92f2d37a2..9c336e483 100644 --- a/docs/plans/2026-10-17-json-tree-index.md +++ b/docs/plans/2026-10-17-json-tree-index.md @@ -496,7 +496,7 @@ Two of the three items that gate a release are closed here; the third, the final - **Planner fragility (gate 2).** `eql_v4.json_tree_query` is plpgsql with `COST 1000` so that a bitmap scan over the GIN index beats a sequential scan even when the documents are TOASTed and the heap looks like a few pages. That was observed in the benches, at 10,000 and 100,000 rows, on one PostgreSQL version, and nothing held it. `postgres_the_planner_picks_the_indexes_without_help` in the encryption suite now does: on a flat table (1,000 four-node documents, every row inline) and a nested table (300 documents of about 220 nodes, every row TOASTed, both regimes pinned by `pg_column_size`), with every planner setting at its default and after `ANALYZE`, containment plans as a bitmap scan over the GIN index and a range at an ordered path as a scan of the functional btree on `ord_term(doc -> selector)`. A control sets the function's cost to 1 for one transaction and shows the planner taking the sequential scan on the nested table, so the assertion is known to be able to fail. The suite runs in `test-eql.yml` in a new `rust-encryption-postgres` job, a matrix over the supported PostgreSQL versions (17 on a pull request, 14 to 17 on a push, like the SQLx jobs); it was a step of the single-version `rust-crates` job. Measured locally on 14, 15, 16 and 17 before the commit. - **Leakage (gate 3).** `packages/eql/docs/reference/json-tree-leakage.md` is the one page: what a stored row shows at rest, field by field; what each query type hands the database; what is authenticated; the side-by-side with v3 from section 6 above, brought up to date (no `g`, position-keyed `p`, ordering opt-in) and corrected where section 6 was wrong (the entry order; see the note under its table); and the two options that change the picture. It says what a token is and that it is column-wide and does not expire, which is the sentence a customer most needs before sending one. - **Open for the owner: the stored entry order (2026-10-23, review of #1336).** `sv` and `q` hold the nodes in the walk's order, depth-first, an object's members in the sorted order of their keys (serde_json's `Map` without `preserve_order`; a build that unifies that feature on would store insertion order instead, which leaks the application's field order). Across rows this reveals each row's shape, its parent relation, every array's length and the alphabetical order of sibling key names, which partly undoes what the keyed rank (decision 11) hides from a token holder. The leakage page says so. Closing it is a format change, not a documentation one: a stored order that does not follow the tree (a document-keyed permutation, say, with `j` the permuted position), a rebuild that does not read the tree from the order, and the same for the pairs of `q`, whose nesting would still group array elements. Not taken here. -- **Taken (2026-10-24, #1355, PR PR_A): an extracted entry is checked to be the one asked for.** Until then `JsonNode` and `JsonSubtree` opened each entry under the `s` and `j` that arrived beside it, and under the data key the header that arrived with it named. The entry's AAD (`entry_aad` in `packages/stack-encrypt/src/json/derive.rs`) binds the keyset, the column's context, the data key's tag, the selector and the ordinal, and nothing that identifies a row. So a decrypted value was always one sealed in that column, but not necessarily the one at the path asked for, nor one from the row the result puts it in: `->` on `$.salary` could be answered with the row's `$.bonus`, with the sealed root (which opens to the whole document), or with another row's `$.salary` returned together with that row's header. Now a row opens only through what the client derived for the read. `JsonSelector::open(node)` refuses an entry whose `s` is not the selector (another path's entry, and the sealed root, whose selector is the root's, in place of a node) before any key is retrieved. `JsonSubtreeKey::open(rows)` refuses a row that does not fit the key's path: a node a gated walk reaches must carry the path's selector at depth `0`, and the position the path names when its last step is one; every other row must carry the selector of its parent's path extended by its own decrypted label (a key, or the wildcard for an element), which the key derives through the keyset's index-key PRF it now holds; and the rows of each subtree must be consecutive in the document, which is what a subtree is in a depth-first `sv`, so a row out of place, a dropped row and an orphan are refused. A reported depth `d` that disagrees is refused too; the rows are placed by their positions and child counts, not by `d`. The refusal is a new `Error::JsonWrongEntry`, not `Aead` (each entry is genuine) and not `OpenedType` (the entry is not of an unexpected shape; it answers another question). `eql-bindings` follows: `JsonTreeExtracted::from_selector(&selector, entry)` is how an entry `->` returned opens, and `JsonTreeEntry` no longer opens on its own. Still not covered: another row's `$.salary` with that row's header opens, by design (two values sealed under the same context are interchangeable between rows, ADR 0010 in `packages/stack-encrypt/docs/adr`, and row provenance is the database's word); and an array position before the last step of a path reads as the wildcard to every selector, so a walk's rows for `$.a[1].b` can be another element's `b` in the same row. Until the ungated walk returns the node it starts from (#1356), a walk with no gates is also not checked for which node at the path's selector it descends from, which matters only for a token chained through array positions (`$.l[3]`), whose children share every other element's selectors. +- **Taken (2026-10-24, #1355, PR #1358): an extracted entry is checked to be the one asked for.** Until then `JsonNode` and `JsonSubtree` opened each entry under the `s` and `j` that arrived beside it, and under the data key the header that arrived with it named. The entry's AAD (`entry_aad` in `packages/stack-encrypt/src/json/derive.rs`) binds the keyset, the column's context, the data key's tag, the selector and the ordinal, and nothing that identifies a row. So a decrypted value was always one sealed in that column, but not necessarily the one at the path asked for, nor one from the row the result puts it in: `->` on `$.salary` could be answered with the row's `$.bonus`, with the sealed root (which opens to the whole document), or with another row's `$.salary` returned together with that row's header. Now a row opens only through what the client derived for the read. `JsonSelector::open(node)` refuses an entry whose `s` is not the selector (another path's entry, and the sealed root, whose selector is the root's, in place of a node) before any key is retrieved. `JsonSubtreeKey::open(rows)` refuses a row that does not fit the key's path: a node a gated walk reaches must carry the path's selector at depth `0`, and the position the path names when its last step is one; every other row must carry the selector of its parent's path extended by its own decrypted label (a key, or the wildcard for an element), which the key derives through the keyset's index-key PRF it now holds; and the rows of each subtree must be consecutive in the document, which is what a subtree is in a depth-first `sv`, so a row out of place, a dropped row and an orphan are refused. A reported depth `d` that disagrees is refused too; the rows are placed by their positions and child counts, not by `d`. The refusal is a new `Error::JsonWrongEntry`, not `Aead` (each entry is genuine) and not `OpenedType` (the entry is not of an unexpected shape; it answers another question). `eql-bindings` follows: `JsonTreeExtracted::from_selector(&selector, entry)` is how an entry `->` returned opens, and `JsonTreeEntry` no longer opens on its own. Still not covered: another row's `$.salary` with that row's header opens, by design (two values sealed under the same context are interchangeable between rows, ADR 0010 in `packages/stack-encrypt/docs/adr`, and row provenance is the database's word); and an array position before the last step of a path reads as the wildcard to every selector, so a walk's rows for `$.a[1].b` can be another element's `b` in the same row. Until the ungated walk returns the node it starts from (#1356), a walk with no gates is also not checked for which node at the path's selector it descends from, which matters only for a token chained through array positions (`$.l[3]`), whose children share every other element's selectors. ## Decisions to take before Phase 1 From 9ec5c59f1f9b309a1c79882ffd6c26acbe445e0e Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 15:51:51 +1100 Subject: [PATCH 3/5] fix(stack-encrypt): refuse a json_tree walk cut short as the wrong entry A walk's rows that stopped before a node's authenticated child count was met were refused as `OpenedType` ("a complete JSON subtree"), while the same row left out between two others was `JsonWrongEntry`. Which error the client raised for a dropped row depended on where the database dropped it. The SQL walk never returns part of a subtree: it returns the node's whole subtree, or past its budget the sealed root alone, and an open root has no budget. So rows that run out early are always the database leaving a row out, and `rebuild_walked` now refuses them the same way it refuses a gap. `open_document` keeps `OpenedType` for a stored document cut short, which is a stored container's shape (ADR 0010), not a read's answer. Tests: the last row of a gated walk's subtree and of a walk without gates, left out; both failed with `OpenedType` before this change. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- packages/stack-encrypt/src/cipher.rs | 5 ++-- packages/stack-encrypt/src/json/derive.rs | 10 +++++-- .../stack-encrypt/tests/json_read_checks.rs | 27 +++++++++++++++++++ 3 files changed, 38 insertions(+), 4 deletions(-) diff --git a/packages/stack-encrypt/src/cipher.rs b/packages/stack-encrypt/src/cipher.rs index 3ebe1d352..327da2785 100644 --- a/packages/stack-encrypt/src/cipher.rs +++ b/packages/stack-encrypt/src/cipher.rs @@ -177,8 +177,9 @@ pub enum Error { /// the one the read asked for: `->` returned the entry of another path, /// or the sealed root in place of a node, or a subtree walk returned a /// row that is not where the walk's key says its rows are (a node of - /// another path, a row out of its place in the document, or a row - /// whose parent is not among the rows). Refused by the client, which + /// another path, a row out of its place in the document, a row whose + /// parent is not among the rows, or rows that stop before a node's + /// subtree is complete). Refused by the client, which /// derived the path's selector itself; see /// [`JsonSelector::open`](crate::json::JsonSelector::open) and /// [`JsonSubtreeKey::open`](crate::json::JsonSubtreeKey::open). diff --git a/packages/stack-encrypt/src/json/derive.rs b/packages/stack-encrypt/src/json/derive.rs index 5af1160dc..e4c986b07 100644 --- a/packages/stack-encrypt/src/json/derive.rs +++ b/packages/stack-encrypt/src/json/derive.rs @@ -1471,7 +1471,9 @@ fn child_selector_bytes(parent: &Protected>, label: &Label) -> Option, cursor: &mut u64| { + // The walk returns a node's whole subtree and stops at no budget + // short of it (the SQL falls back to the sealed root instead), so + // rows that run out before a child count is met are a row left out + // at the end, the same refusal as one left out in the middle. let row = rest .next() - .ok_or_else(|| shape("a complete JSON subtree"))?; + .ok_or_else(|| wrong("the next node of the walked subtree"))?; let child_bytes = child_selector_bytes(bytes, &row.node.label) .ok_or_else(|| wrong("the next node of the walked subtree"))?; if row.j != *cursor diff --git a/packages/stack-encrypt/tests/json_read_checks.rs b/packages/stack-encrypt/tests/json_read_checks.rs index 963b37ac8..39e123e12 100644 --- a/packages/stack-encrypt/tests/json_read_checks.rs +++ b/packages/stack-encrypt/tests/json_read_checks.rs @@ -438,6 +438,19 @@ mod given_the_rows_of_a_walk_without_gates { assert_eq!(opened.unwrap(), value, "the whole document"); } + #[tokio::test] + async fn a_subtree_cut_short_is_refused_as_the_wrong_entry() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let nested = key(&cipher, "$.nested").await; + // The last row is the second element's `x` (8): without it the + // element says it has a member the rows do not hold. + let mut rows = walk(&doc, &nested); + assert_eq!(rows.pop().map(|n| n.ordinal()), Some(8), "x, the last row"); + let refused = open_value(&cipher, nested.open(rows)).await; + assertions::wrong_entry(refused, NEXT); + } + #[tokio::test] async fn rows_with_no_reported_depth_are_placed_by_the_document() { let (cipher, _) = cipher().await; @@ -516,6 +529,20 @@ mod given_the_rows_of_a_gated_walk { assertions::wrong_entry(refused, REACHED); } + #[tokio::test] + async fn a_subtree_cut_short_is_refused_as_the_wrong_entry() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let second = key(&cipher, "$.nested.list[1]").await; + // The element (7) says it has one member, and the database leaves + // that member (8) out: the same dropped row as a gap, at the end of + // the rows instead of between two of them. + let mut rows = walk(&doc, &second); + assert_eq!(rows.pop().map(|n| n.ordinal()), Some(8), "x, the last row"); + let refused = open_value(&cipher, second.open(rows)).await; + assertions::wrong_entry(refused, "the next node of the walked subtree"); + } + #[tokio::test] async fn the_sealed_root_among_walked_rows_is_refused() { let (cipher, _) = cipher().await; From aa591f2402bd010150a5a808e5f32912db517673 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 15:53:05 +1100 Subject: [PATCH 4/5] fix(stack-encrypt): refuse a json_tree walk that leaves out an array's first element `json_tree_subtree` (a key with no gates) returns the descendants of the token's node but not the node, so nothing before the first row bounds where the children begin. The consecutive-row check caught a child left out between two others, but not the first: for `{"arr": [10, 20, 30]}` a database that dropped element 0's row had `$.arr` open as `[20, 30]`, every element moved down one position, with no error. An array's children are its elements from 0, so `place` now requires the nth row at depth 1 that is an element to be element n. A member's place is a keyed rank the client does not hold, so a member left out, and the last children of either kind, are still undetected; the rustdoc on `JsonSubtreeKey::open` says so. Both close once the walk returns the node it starts from (#1356), whose child count bounds its children. Tests: the first element left out, opened as a value and as a list (both opened before this change), and a pin of the trailing case that stays open. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- packages/stack-encrypt/src/json/derive.rs | 14 +++++++- packages/stack-encrypt/src/json/query.rs | 16 ++++++--- .../stack-encrypt/tests/json_read_checks.rs | 34 +++++++++++++++++++ 3 files changed, 58 insertions(+), 6 deletions(-) diff --git a/packages/stack-encrypt/src/json/derive.rs b/packages/stack-encrypt/src/json/derive.rs index e4c986b07..d50ab1f5f 100644 --- a/packages/stack-encrypt/src/json/derive.rs +++ b/packages/stack-encrypt/src/json/derive.rs @@ -1382,7 +1382,8 @@ fn open_subtree<'a, K>( /// last step is one. Without gates, the rows are the children of the /// path's node and their subtrees, side by side: each child carries the /// selector of the path extended by its label, at depth `1`, right after -/// the subtree before it. +/// the subtree before it, and an element at the position after the +/// elements before it, from `0`. fn place( rows: Vec, key: &JsonSubtreeKey, @@ -1399,8 +1400,19 @@ fn place( let (bytes, depth) = if below { let bytes = child_selector_bytes(&at, &top.node.label) .ok_or_else(|| wrong("a child of the walked node"))?; + // The walk does not return the node it starts from, so no row + // bounds where the children begin; an element's position does. + // The children of an array are its elements from `0`, so the + // `n`th row at depth 1 must be element `n`: a database that + // leaves out the first element would otherwise move every + // element down a position. A member's place is a keyed rank + // the client does not hold, so a member left out, and the last + // children of either, are still the database's word. + let out_of_place = + matches!(top.node.label, Label::Index(i) if i != forest.len() as u64); if top.d.is_some_and(|d| d != 1) || next.is_some_and(|j| j != top.j) + || out_of_place || key.selectors.of(bytes.clone())? != top.s { return Err(wrong("a child of the walked node")); diff --git a/packages/stack-encrypt/src/json/query.rs b/packages/stack-encrypt/src/json/query.rs index 1600fbe47..8ca017004 100644 --- a/packages/stack-encrypt/src/json/query.rs +++ b/packages/stack-encrypt/src/json/query.rs @@ -437,12 +437,18 @@ impl JsonSubtreeKey { /// no entry between, so that each node's rows are exactly its subtree. /// A row whose parent is not among the rows is refused. /// + /// With no gates the walk does not return the node it starts from, so + /// its children are checked against each other: they follow one + /// another with no gap, and an array's start at element `0`. + /// /// Not covered: which row of the column the entries come from (ADR - /// 0010), and, until the walk returns the node it starts from, which - /// node at the path's selector a walk with no gates descends from. An - /// array position before the last step of a path reads as the wildcard - /// to the selectors, so another element's subtree at the same depth - /// passes too. + /// 0010), and, until the walk returns the node it starts from (#1356), + /// two things about a walk with no gates: which node at the path's + /// selector it descends from, and whether children were left out after + /// the last row, or members anywhere (a member's place is a keyed rank + /// the client does not hold). An array position before the last step + /// of a path reads as the wildcard to the selectors, so another + /// element's subtree at the same depth passes too. pub fn open(&self, rows: Vec) -> JsonSubtree { JsonSubtree::keyed(rows, self.clone()) } diff --git a/packages/stack-encrypt/tests/json_read_checks.rs b/packages/stack-encrypt/tests/json_read_checks.rs index 39e123e12..53f0f626a 100644 --- a/packages/stack-encrypt/tests/json_read_checks.rs +++ b/packages/stack-encrypt/tests/json_read_checks.rs @@ -451,6 +451,40 @@ mod given_the_rows_of_a_walk_without_gates { assertions::wrong_entry(refused, NEXT); } + #[tokio::test] + async fn a_leading_element_left_out_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &json!({"arr": [10, 20, 30]})).await; + let arr = key(&cipher, "$.arr").await; + assert!(arr.gates().is_empty(), "a chain head's token"); + // The walk does not return the node it starts from, so nothing + // before the first row bounds where the rows begin. Leaving out the + // first element would open as `[20, 30]`, every element moved down + // one position. + let rows = walk(&doc, &arr); + let first = rows[0].ordinal(); + let rows: Vec = rows.into_iter().filter(|n| n.ordinal() != first).collect(); + assertions::wrong_entry(open_value(&cipher, arr.open(rows.clone())).await, CHILD); + assertions::wrong_entry(open_list(&cipher, arr.open(rows)).await, CHILD); + } + + #[tokio::test] + async fn trailing_children_left_out_are_not_detected_by_a_walk_without_gates() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &json!({"arr": [10, 20, 30]})).await; + let arr = key(&cipher, "$.arr").await; + // What the design leaves open until the walk returns the node it + // starts from (#1356): the node's child count is in the node's own + // entry, so the rows cannot say how many children follow the last + // one returned. The database withholding the last element opens as + // the shorter array. A member left out of an object is the same, + // wherever it sits, since nothing the client holds ranks the keys. + let mut rows = walk(&doc, &arr); + rows.pop(); + let opened = open_value(&cipher, arr.open(rows)).await; + assert_eq!(opened.unwrap(), json!([10, 20]), "the elements before it"); + } + #[tokio::test] async fn rows_with_no_reported_depth_are_placed_by_the_document() { let (cipher, _) = cipher().await; From efc6bba231d6989159cef168176cafe346cae665 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sun, 25 Oct 2026 15:53:43 +1100 Subject: [PATCH 5/5] docs(eql): say what a json_tree walk with no gates still does not check The leakage page's "What is authenticated" said a dropped row is refused and that each decrypted value is at the path asked for. For a walk with no gates neither held in full: the walk does not return the node it starts from, so children left out after the last row (and members left out anywhere) are not detected, and for a token chained through array positions (`$.l[3]`) the children can be another element's. The second was in the rustdoc and the design record but not on the page customers read. The page now lists both under what the checks do not cover, with the issue that closes them (#1356), and says what is checked for that walk: no gap between children, an array starting at element 0, and no row left out of a node's subtree. The design record's bullet says the same. Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu --- docs/plans/2026-10-17-json-tree-index.md | 2 +- packages/eql/docs/reference/json-tree-leakage.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/plans/2026-10-17-json-tree-index.md b/docs/plans/2026-10-17-json-tree-index.md index 9c336e483..8bb755ae6 100644 --- a/docs/plans/2026-10-17-json-tree-index.md +++ b/docs/plans/2026-10-17-json-tree-index.md @@ -496,7 +496,7 @@ Two of the three items that gate a release are closed here; the third, the final - **Planner fragility (gate 2).** `eql_v4.json_tree_query` is plpgsql with `COST 1000` so that a bitmap scan over the GIN index beats a sequential scan even when the documents are TOASTed and the heap looks like a few pages. That was observed in the benches, at 10,000 and 100,000 rows, on one PostgreSQL version, and nothing held it. `postgres_the_planner_picks_the_indexes_without_help` in the encryption suite now does: on a flat table (1,000 four-node documents, every row inline) and a nested table (300 documents of about 220 nodes, every row TOASTed, both regimes pinned by `pg_column_size`), with every planner setting at its default and after `ANALYZE`, containment plans as a bitmap scan over the GIN index and a range at an ordered path as a scan of the functional btree on `ord_term(doc -> selector)`. A control sets the function's cost to 1 for one transaction and shows the planner taking the sequential scan on the nested table, so the assertion is known to be able to fail. The suite runs in `test-eql.yml` in a new `rust-encryption-postgres` job, a matrix over the supported PostgreSQL versions (17 on a pull request, 14 to 17 on a push, like the SQLx jobs); it was a step of the single-version `rust-crates` job. Measured locally on 14, 15, 16 and 17 before the commit. - **Leakage (gate 3).** `packages/eql/docs/reference/json-tree-leakage.md` is the one page: what a stored row shows at rest, field by field; what each query type hands the database; what is authenticated; the side-by-side with v3 from section 6 above, brought up to date (no `g`, position-keyed `p`, ordering opt-in) and corrected where section 6 was wrong (the entry order; see the note under its table); and the two options that change the picture. It says what a token is and that it is column-wide and does not expire, which is the sentence a customer most needs before sending one. - **Open for the owner: the stored entry order (2026-10-23, review of #1336).** `sv` and `q` hold the nodes in the walk's order, depth-first, an object's members in the sorted order of their keys (serde_json's `Map` without `preserve_order`; a build that unifies that feature on would store insertion order instead, which leaks the application's field order). Across rows this reveals each row's shape, its parent relation, every array's length and the alphabetical order of sibling key names, which partly undoes what the keyed rank (decision 11) hides from a token holder. The leakage page says so. Closing it is a format change, not a documentation one: a stored order that does not follow the tree (a document-keyed permutation, say, with `j` the permuted position), a rebuild that does not read the tree from the order, and the same for the pairs of `q`, whose nesting would still group array elements. Not taken here. -- **Taken (2026-10-24, #1355, PR #1358): an extracted entry is checked to be the one asked for.** Until then `JsonNode` and `JsonSubtree` opened each entry under the `s` and `j` that arrived beside it, and under the data key the header that arrived with it named. The entry's AAD (`entry_aad` in `packages/stack-encrypt/src/json/derive.rs`) binds the keyset, the column's context, the data key's tag, the selector and the ordinal, and nothing that identifies a row. So a decrypted value was always one sealed in that column, but not necessarily the one at the path asked for, nor one from the row the result puts it in: `->` on `$.salary` could be answered with the row's `$.bonus`, with the sealed root (which opens to the whole document), or with another row's `$.salary` returned together with that row's header. Now a row opens only through what the client derived for the read. `JsonSelector::open(node)` refuses an entry whose `s` is not the selector (another path's entry, and the sealed root, whose selector is the root's, in place of a node) before any key is retrieved. `JsonSubtreeKey::open(rows)` refuses a row that does not fit the key's path: a node a gated walk reaches must carry the path's selector at depth `0`, and the position the path names when its last step is one; every other row must carry the selector of its parent's path extended by its own decrypted label (a key, or the wildcard for an element), which the key derives through the keyset's index-key PRF it now holds; and the rows of each subtree must be consecutive in the document, which is what a subtree is in a depth-first `sv`, so a row out of place, a dropped row and an orphan are refused. A reported depth `d` that disagrees is refused too; the rows are placed by their positions and child counts, not by `d`. The refusal is a new `Error::JsonWrongEntry`, not `Aead` (each entry is genuine) and not `OpenedType` (the entry is not of an unexpected shape; it answers another question). `eql-bindings` follows: `JsonTreeExtracted::from_selector(&selector, entry)` is how an entry `->` returned opens, and `JsonTreeEntry` no longer opens on its own. Still not covered: another row's `$.salary` with that row's header opens, by design (two values sealed under the same context are interchangeable between rows, ADR 0010 in `packages/stack-encrypt/docs/adr`, and row provenance is the database's word); and an array position before the last step of a path reads as the wildcard to every selector, so a walk's rows for `$.a[1].b` can be another element's `b` in the same row. Until the ungated walk returns the node it starts from (#1356), a walk with no gates is also not checked for which node at the path's selector it descends from, which matters only for a token chained through array positions (`$.l[3]`), whose children share every other element's selectors. +- **Taken (2026-10-24, #1355, PR #1358): an extracted entry is checked to be the one asked for.** Until then `JsonNode` and `JsonSubtree` opened each entry under the `s` and `j` that arrived beside it, and under the data key the header that arrived with it named. The entry's AAD (`entry_aad` in `packages/stack-encrypt/src/json/derive.rs`) binds the keyset, the column's context, the data key's tag, the selector and the ordinal, and nothing that identifies a row. So a decrypted value was always one sealed in that column, but not necessarily the one at the path asked for, nor one from the row the result puts it in: `->` on `$.salary` could be answered with the row's `$.bonus`, with the sealed root (which opens to the whole document), or with another row's `$.salary` returned together with that row's header. Now a row opens only through what the client derived for the read. `JsonSelector::open(node)` refuses an entry whose `s` is not the selector (another path's entry, and the sealed root, whose selector is the root's, in place of a node) before any key is retrieved. `JsonSubtreeKey::open(rows)` refuses a row that does not fit the key's path: a node a gated walk reaches must carry the path's selector at depth `0`, and the position the path names when its last step is one; every other row must carry the selector of its parent's path extended by its own decrypted label (a key, or the wildcard for an element), which the key derives through the keyset's index-key PRF it now holds; and the rows of each subtree must be consecutive in the document, which is what a subtree is in a depth-first `sv`, so a row out of place, a dropped row and an orphan are refused. A reported depth `d` that disagrees is refused too; the rows are placed by their positions and child counts, not by `d`. The refusal is a new `Error::JsonWrongEntry`, not `Aead` (each entry is genuine) and not `OpenedType` (the entry is not of an unexpected shape; it answers another question). `eql-bindings` follows: `JsonTreeExtracted::from_selector(&selector, entry)` is how an entry `->` returned opens, and `JsonTreeEntry` no longer opens on its own. Still not covered: another row's `$.salary` with that row's header opens, by design (two values sealed under the same context are interchangeable between rows, ADR 0010 in `packages/stack-encrypt/docs/adr`, and row provenance is the database's word); and an array position before the last step of a path reads as the wildcard to every selector, so a walk's rows for `$.a[1].b` can be another element's `b` in the same row. Until the ungated walk returns the node it starts from (#1356), a walk with no gates is also not checked for which node at the path's selector it descends from, which matters only for a token chained through array positions (`$.l[3]`), whose children share every other element's selectors; nor for children left out after the last row it returns, or members left out anywhere, since the node's child count is in the node's own entry and a member's place is a keyed rank. An array's elements are checked to start at `0` (review of PR #1358: leaving out element 0 opened `[10, 20, 30]` as `[20, 30]`), and rows that stop short of a node's subtree are `JsonWrongEntry` like a gap. ## Decisions to take before Phase 1 diff --git a/packages/eql/docs/reference/json-tree-leakage.md b/packages/eql/docs/reference/json-tree-leakage.md index 265aa6fd6..f70901e7b 100644 --- a/packages/eql/docs/reference/json-tree-leakage.md +++ b/packages/eql/docs/reference/json-tree-leakage.md @@ -42,9 +42,9 @@ Only the ciphertext `c`. Opening an entry verifies that it was sealed under this The consequence to design around: tampering with those fields changes **which rows a query returns, where a walk stops, and which entries of a row an extraction or a walk hands back**. An entry the database returns from `->` or a walk carries its own `s` and `j`, and the header it carries names the data key, so a decrypted value is always one that was sealed in this column. The client also checks that it is the entry it asked for, from what it derived for the query, and refuses anything else (`JsonWrongEntry`): - **`->`.** The entry must carry the selector the client derived for the path. Another path's entry, such as the row's `$.bonus` returned for `$.salary`, is refused, and so is the sealed root returned for a field (it opens to the whole document); the root is accepted only when `$` itself was asked for. -- **A walk.** A node a gated walk reaches must carry the path's selector, and the position the path ends in, if it ends in one. Every other row must carry the selector of its parent's path extended by its own key (or the wildcard, for an array element), which the client derives from the decrypted key, and a subtree's rows must follow each other in the document with no gap. A row of another path, a dropped or reordered row, and a row whose parent is not among the rows are refused. +- **A walk.** A node a gated walk reaches must carry the path's selector, and the position the path ends in, if it ends in one. Every other row must carry the selector of its parent's path extended by its own key (or the wildcard, for an array element), which the client derives from the decrypted key, and a subtree's rows must follow each other in the document with no gap, through to the last row the node's child count calls for. A row of another path, a reordered row, a row whose parent is not among the rows, and a row left out of a node's subtree are refused. -What the checks do not cover: **which row an entry comes from**. Nothing in an entry names its row, so another row's `$.salary`, returned with that row's header, opens; that is the rule every EQL stored value follows (ADR 0010). And **a position before the last step of a path**: every element of an array shares its selector, so a walk for `$.orders[1].items` can be answered with `$.orders[0].items` of the same row. A query's result is not integrity-protected as a whole: each decrypted value is genuine and at the path asked for, but which row it came from is the database's word. +What the checks do not cover: **which row an entry comes from**. Nothing in an entry names its row, so another row's `$.salary`, returned with that row's header, opens; that is the rule every EQL stored value follows (ADR 0010). And **a position before the last step of a path**: every element of an array shares its selector, so a walk for `$.orders[1].items` can be answered with `$.orders[0].items` of the same row. And, for a walk with no gates (`json_tree_subtree`, whose path is a top-level field, or the root when the root is not sealed, followed only by array positions), **the node the walk starts from**, which it does not return: its children are checked against each other, so a gap between two of them and an array that does not start at element `0` are refused, but children left out after the last one returned, and members left out anywhere, are not (a member's place is a keyed rank the client does not hold), and for a path that ends in a position (`$.orders[3]`) the children can be another element's, since every element's children share their selectors. Both close when the walk returns the node it starts from, whose child count bounds its children (#1356). A query's result is not integrity-protected as a whole: each decrypted value is genuine and sealed at the path asked for, read as a selector reads it, but which row it came from, and which of the same-selector nodes above, is the database's word. ## Compared with EQL v3 (`public.eql_v3_json_search`)