diff --git a/.changeset/eql-json-tree-extracted-entry-check.md b/.changeset/eql-json-tree-extracted-entry-check.md new file mode 100644 index 000000000..04662ee6c --- /dev/null +++ b/.changeset/eql-json-tree-extracted-entry-check.md @@ -0,0 +1,5 @@ +--- +"@cipherstash/eql": patch +--- + +EQL v4 `json_tree` (unreleased): the client now checks that what `->` and the subtree walks return is what it asked for. Every entry of a stored document is sealed under its own path selector and position, so any entry of the row opened, and a database could answer `doc -> s($.salary)` with the row's `$.bonus`, or with the sealed root (the whole document), or splice other entries into a walk. In the `eql-bindings` crate an entry `->` returned now opens through the selector that asked for it, `JsonTreeExtracted::from_selector(&selector, entry)`, which refuses an entry carrying any other selector; `JsonTreeEntry` no longer opens on its own. A walk's rows, opened through `JsonTreeSubtree::from_key(&key, rows)` as before, are each checked against the key's path: the node a gated walk reaches must carry the path's selector and position, every other row the selector of its parent's path and its own label, and the rows must follow each other in the document with no gap. A refusal is `stack_encrypt::Error::JsonWrongEntry`. Not checked, by design: which row of the column an entry comes from (ADR 0010). The SQL is unchanged. diff --git a/docs/plans/2026-10-17-json-tree-index.md b/docs/plans/2026-10-17-json-tree-index.md index 41b1e06b7..9d963b8d6 100644 --- a/docs/plans/2026-10-17-json-tree-index.md +++ b/docs/plans/2026-10-17-json-tree-index.md @@ -496,7 +496,7 @@ Two of the three items that gate a release are closed here; the third, the final - **Planner fragility (gate 2).** `eql_v4.json_tree_query` is plpgsql with `COST 1000` so that a bitmap scan over the GIN index beats a sequential scan even when the documents are TOASTed and the heap looks like a few pages. That was observed in the benches, at 10,000 and 100,000 rows, on one PostgreSQL version, and nothing held it. `postgres_the_planner_picks_the_indexes_without_help` in the encryption suite now does: on a flat table (1,000 four-node documents, every row inline) and a nested table (300 documents of about 220 nodes, every row TOASTed, both regimes pinned by `pg_column_size`), with every planner setting at its default and after `ANALYZE`, containment plans as a bitmap scan over the GIN index and a range at an ordered path as a scan of the functional btree on `ord_term(doc -> selector)`. A control sets the function's cost to 1 for one transaction and shows the planner taking the sequential scan on the nested table, so the assertion is known to be able to fail. The suite runs in `test-eql.yml` in a new `rust-encryption-postgres` job, a matrix over the supported PostgreSQL versions (17 on a pull request, 14 to 17 on a push, like the SQLx jobs); it was a step of the single-version `rust-crates` job. Measured locally on 14, 15, 16 and 17 before the commit. - **Leakage (gate 3).** `packages/eql/docs/reference/json-tree-leakage.md` is the one page: what a stored row shows at rest, field by field; what each query type hands the database; what is authenticated; the side-by-side with v3 from section 6 above, brought up to date (no `g`, position-keyed `p`, ordering opt-in) and corrected where section 6 was wrong (the entry order; see the note under its table); and the two options that change the picture. It says what a token is and that it is column-wide and does not expire, which is the sentence a customer most needs before sending one. - **Open for the owner: the stored entry order (2026-10-23, review of #1336).** `sv` and `q` hold the nodes in the walk's order, depth-first, an object's members in the sorted order of their keys (serde_json's `Map` without `preserve_order`; a build that unifies that feature on would store insertion order instead, which leaks the application's field order). Across rows this reveals each row's shape, its parent relation, every array's length and the alphabetical order of sibling key names, which partly undoes what the keyed rank (decision 11) hides from a token holder. The leakage page says so. Closing it is a format change, not a documentation one: a stored order that does not follow the tree (a document-keyed permutation, say, with `j` the permuted position), a rebuild that does not read the tree from the order, and the same for the pairs of `q`, whose nesting would still group array elements. Not taken here. -- **Open for the owner: an extracted entry opens under the selector the server returns with it.** `JsonNode` and `JsonSubtree` open each entry under the `s` and `j` that arrive beside it, and under the data key the header that arrives with it names. The entry's AAD (`entry_aad` in `packages/stack-encrypt/src/json/derive.rs`) binds the keyset, the column's context, the data key's tag, the selector and the ordinal, and nothing that identifies a row. So a decrypted value is always one sealed in that column, but not necessarily the one at the path asked for, nor one from the row the result puts it in: `->` on `$.salary` can be answered with the row's `$.bonus`, with the sealed root (which opens to the whole document), or with another row's `$.salary` returned together with that row's header. The last is by design: two values sealed under the same context are interchangeable between rows (ADR 0010 in `packages/stack-encrypt/docs/adr`), and row provenance is the database's word. The leakage page states this under "What is authenticated". Checking the returned `s` against the selector the client derived would close the path substitution for `->`, though not the row one; a walk's rows would need each child's selector checked against its parent's path. +- **Taken (2026-10-24, #1355, PR #1358): an extracted entry is checked to be the one asked for.** Until then `JsonNode` and `JsonSubtree` opened each entry under the `s` and `j` that arrived beside it, and under the data key the header that arrived with it named. The entry's AAD (`entry_aad` in `packages/stack-encrypt/src/json/derive.rs`) binds the keyset, the column's context, the data key's tag, the selector and the ordinal, and nothing that identifies a row. So a decrypted value was always one sealed in that column, but not necessarily the one at the path asked for, nor one from the row the result puts it in: `->` on `$.salary` could be answered with the row's `$.bonus`, with the sealed root (which opens to the whole document), or with another row's `$.salary` returned together with that row's header. Now a row opens only through what the client derived for the read. `JsonSelector::open(node)` refuses an entry whose `s` is not the selector (another path's entry, and the sealed root, whose selector is the root's, in place of a node) before any key is retrieved. `JsonSubtreeKey::open(rows)` refuses a row that does not fit the key's path: a node a gated walk reaches must carry the path's selector at depth `0`, and the position the path names when its last step is one; every other row must carry the selector of its parent's path extended by its own decrypted label (a key, or the wildcard for an element), which the key derives through the keyset's index-key PRF it now holds; and the rows of each subtree must be consecutive in the document, which is what a subtree is in a depth-first `sv`, so a row out of place, a dropped row and an orphan are refused. A reported depth `d` that disagrees is refused too; the rows are placed by their positions and child counts, not by `d`. The refusal is a new `Error::JsonWrongEntry`, not `Aead` (each entry is genuine) and not `OpenedType` (the entry is not of an unexpected shape; it answers another question). `eql-bindings` follows: `JsonTreeExtracted::from_selector(&selector, entry)` is how an entry `->` returned opens, and `JsonTreeEntry` no longer opens on its own. Still not covered: another row's `$.salary` with that row's header opens, by design (two values sealed under the same context are interchangeable between rows, ADR 0010 in `packages/stack-encrypt/docs/adr`, and row provenance is the database's word); and an array position before the last step of a path reads as the wildcard to every selector, so a walk's rows for `$.a[1].b` can be another element's `b` in the same row. Until the ungated walk returns the node it starts from (#1356), a walk with no gates is also not checked for which node at the path's selector it descends from, which matters only for a token chained through array positions (`$.l[3]`), whose children share every other element's selectors; nor for children left out after the last row it returns, or members left out anywhere, since the node's child count is in the node's own entry and a member's place is a keyed rank. An array's elements are checked to start at `0` (review of PR #1358: leaving out element 0 opened `[10, 20, 30]` as `[20, 30]`), and rows that stop short of a node's subtree are `JsonWrongEntry` like a gap. ## Decisions to take before Phase 1 diff --git a/packages/eql/crates/eql-bindings/CHANGELOG.md b/packages/eql/crates/eql-bindings/CHANGELOG.md index 2622f6fa7..69ce58852 100644 --- a/packages/eql/crates/eql-bindings/CHANGELOG.md +++ b/packages/eql/crates/eql-bindings/CHANGELOG.md @@ -20,8 +20,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 opens at the path the walk's key was derived from: the rows are built with `JsonTreeSubtree::from_key(&key, rows)`, and there is no constructor without the key. + An entry `->` returned opens through the selector that asked for it, + `JsonTreeExtracted::from_selector(&selector, entry)`, which refuses an + entry carrying another selector (`stack_encrypt::Error::JsonWrongEntry`), + and a walk's rows through the key that keyed it, which checks each row + against the path the same way; `JsonTreeEntry` does not open on its own. The document is an `EncryptFrom` and `DecryptInto` target, - the entry and the subtree open through `DecryptInto`, and the needle and + the extracted entry and the subtree open through `DecryptInto`, and the needle and the ordering operand convert from Stack Encrypt's `JsonQuery` and `JsonLeafQuery`. `JsonTree::new(options)` is the `json_tree` layout under index options other than the defaults (a `Layout`, so a program diff --git a/packages/eql/crates/eql-bindings/src/v4/json_tree.rs b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs index 26507af85..ba0f77eb7 100644 --- a/packages/eql/crates/eql-bindings/src/v4/json_tree.rs +++ b/packages/eql/crates/eql-bindings/src/v4/json_tree.rs @@ -23,8 +23,8 @@ use schemars::{schema_for, JsonSchema, Schema}; use serde::{Deserialize, Serialize}; use serde_json::Value; use stack_encrypt::json::{ - JsonDocument, JsonLeafQuery, JsonNode, JsonOptions, JsonPath, JsonQuery, JsonSubtree, - JsonSubtreeKey, + JsonDocument, JsonLeafQuery, JsonNode, JsonOptions, JsonPath, JsonQuery, JsonSelector, + JsonSubtree, JsonSubtreeKey, }; use stack_encrypt::target::{AeadContext, ExpectedContext, IndexSpec, Json, Layout}; use stack_encrypt::{ @@ -94,7 +94,13 @@ pub struct JsonTreeDocument { /// `c` the node's own ciphertext; `p` the parent tag below the chain's /// heads, which the walk probes by ordinal; `op` the leaf ordering term. /// `h`, `j`, `d`, `v` and `i` are what the extractors graft onto an entry -/// they return, so it opens on its own. +/// they return, so that it can be opened. +/// +/// An entry a query returned is the database's word and does not open on +/// its own: open the entry `->` returned through the selector that asked +/// for it ([`JsonTreeExtracted::from_selector`]), and a walk's rows through +/// the key that keyed it ([`JsonTreeSubtree::from_key`]). Each checks that +/// the rows are the ones it asked for. #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)] #[serde(deny_unknown_fields)] pub struct JsonTreeEntry { @@ -165,6 +171,42 @@ pub struct JsonTreeLeafQuery { pub op: JsonTreeOrderingTerm, } +/// The entry `col -> selector` returned, with the selector that asked for +/// it: opens to the scalar the entry holds, or to the whole document for a +/// sealed root asked for at `$`. The only constructor, +/// [`from_selector`](Self::from_selector), takes the selector, and opening +/// refuses an entry that does not carry it +/// ([`Error::JsonWrongEntry`](stack_encrypt::Error::JsonWrongEntry)), as +/// [`JsonSelector::open`] does: a database that answers one path with +/// another path's entry, or with the sealed root, returns an entry that +/// would otherwise open. +#[derive(Clone, Debug, PartialEq)] +pub struct JsonTreeExtracted { + entry: JsonTreeEntry, + selector: JsonSelector, +} + +impl JsonTreeExtracted { + /// The entry `col -> selector` returned for `selector`, as the server + /// sent it. + pub fn from_selector(selector: &JsonSelector, entry: JsonTreeEntry) -> Self { + Self { + entry, + selector: *selector, + } + } + + /// The entry. + pub fn entry(&self) -> &JsonTreeEntry { + &self.entry + } + + /// The selector the read asked for. + pub fn selector(&self) -> &JsonSelector { + &self.selector + } +} + /// The rows a subtree walk returned (`eql_v4.json_tree_subtree` or /// `json_tree_subtree_at`), collected with the key that keyed the walk: /// opens to the value under the walked path, or to the list of complete @@ -396,14 +438,18 @@ where } /// An extracted entry opens to the scalar it holds, or to the whole -/// document for a sealed root, under the column it was extracted from. A +/// document for a sealed root asked for at `$`, under the column it was +/// extracted from, once it is shown to carry the selector asked for. A /// container node holds no value of its own; its subtree opens through /// [`JsonTreeSubtree`]. -impl DecryptInto for JsonTreeEntry { +impl DecryptInto for JsonTreeExtracted { type Context = NonEmpty; fn decryption(self, context: Self::Context) -> Decryption { - match self.node() { - Ok(node) => node.decryption(AeadContext::from(context)), + match self.entry.node() { + Ok(node) => self + .selector + .open(node) + .decryption(AeadContext::from(context)), Err(error) => Decryption::failed(error), } } diff --git a/packages/eql/docs/reference/json-tree-leakage.md b/packages/eql/docs/reference/json-tree-leakage.md index bfb6ffab2..f70901e7b 100644 --- a/packages/eql/docs/reference/json-tree-leakage.md +++ b/packages/eql/docs/reference/json-tree-leakage.md @@ -39,7 +39,12 @@ A query hands the database a term derived from the plaintext it asks about, and Only the ciphertext `c`. Opening an entry verifies that it was sealed under this keyset, for this column, under the data key its row's header names (a fresh one for every row), with the selector `s` and the position `j` it is opened with; a ciphertext moved into another row or column, or opened under another selector or position, fails. Nothing in it names a row, so an entry that travels with its own row's header still opens; see below. When the client opens a whole document it takes each entry's position from its place in `sv`, so a moved, swapped or re-labelled entry fails to open. The selectors, the projection, the parent tags, the ordering terms, the header's sealed-root flag, and which entries a row holds are not authenticated, by the same rule every EQL stored value follows (ADR 0010 in `packages/stack-encrypt/docs/adr`). -The consequence to design around: tampering with those fields changes **which rows a query returns, where a walk stops, and which entries of a row an extraction or a walk hands back**. An entry the database returns from `->` or a walk carries its own `s` and `j`, and the client opens it under those, and the header it carries names the data key, so a decrypted value is always one that was sealed in this column, but the client does not check today that it is the entry at the path it asked for, or that it comes from the row the result puts it in: a database that wants to can answer `$.salary` with the same row's `$.bonus`, or with another row's `$.salary`. A query's result is not integrity-protected; each decrypted value is genuine, but where it came from is the database's word. +The consequence to design around: tampering with those fields changes **which rows a query returns, where a walk stops, and which entries of a row an extraction or a walk hands back**. An entry the database returns from `->` or a walk carries its own `s` and `j`, and the header it carries names the data key, so a decrypted value is always one that was sealed in this column. The client also checks that it is the entry it asked for, from what it derived for the query, and refuses anything else (`JsonWrongEntry`): + +- **`->`.** The entry must carry the selector the client derived for the path. Another path's entry, such as the row's `$.bonus` returned for `$.salary`, is refused, and so is the sealed root returned for a field (it opens to the whole document); the root is accepted only when `$` itself was asked for. +- **A walk.** A node a gated walk reaches must carry the path's selector, and the position the path ends in, if it ends in one. Every other row must carry the selector of its parent's path extended by its own key (or the wildcard, for an array element), which the client derives from the decrypted key, and a subtree's rows must follow each other in the document with no gap, through to the last row the node's child count calls for. A row of another path, a reordered row, a row whose parent is not among the rows, and a row left out of a node's subtree are refused. + +What the checks do not cover: **which row an entry comes from**. Nothing in an entry names its row, so another row's `$.salary`, returned with that row's header, opens; that is the rule every EQL stored value follows (ADR 0010). And **a position before the last step of a path**: every element of an array shares its selector, so a walk for `$.orders[1].items` can be answered with `$.orders[0].items` of the same row. And, for a walk with no gates (`json_tree_subtree`, whose path is a top-level field, or the root when the root is not sealed, followed only by array positions), **the node the walk starts from**, which it does not return: its children are checked against each other, so a gap between two of them and an array that does not start at element `0` are refused, but children left out after the last one returned, and members left out anywhere, are not (a member's place is a keyed rank the client does not hold), and for a path that ends in a position (`$.orders[3]`) the children can be another element's, since every element's children share their selectors. Both close when the walk returns the node it starts from, whose child count bounds its children (#1356). A query's result is not integrity-protected as a whole: each decrypted value is genuine and sealed at the path asked for, read as a selector reads it, but which row it came from, and which of the same-selector nodes above, is the database's word. ## Compared with EQL v3 (`public.eql_v3_json_search`) @@ -62,6 +67,6 @@ The consequence to design around: tampering with those fields changes **which ro - **`sealed_root`** (default on). The root entry's ciphertext is the whole document, so a full read is one decryption and the broadest token a query can send is one top-level field's. The costs are that the root entry's length reveals the document's size, the row is larger, and the root's direct members are marked by their missing `p`, which puts each top-level field's node count in plain view. Off, there is no document-sized ciphertext, but every subtree read whose path begins with a key sends the root's token, the whole column's shape. - **`ordered`** (default none). Each path listed carries an ordering term and takes on the `op` row above; a path not listed cannot be range-queried or sorted, and a query that tries is refused by the client before any key loads. Order only the paths your queries sort or range over. -Two disclosures above are properties of the stored format that no option changes today: the depth-first entry order, with members in sorted key order, and the client opening an extracted entry under the selector the database returns with it. Both are open decisions recorded in the design document (`docs/plans/2026-10-17-json-tree-index.md` at the repository root, Phase 4 notes). +One disclosure above is a property of the stored format that no option changes today: the depth-first entry order, with members in sorted key order. It is an open decision recorded in the design document (`docs/plans/2026-10-17-json-tree-index.md` at the repository root, Phase 4 notes) and in #1357. To report a concern with any of this, see `SECURITY.md` at the repository root. diff --git a/packages/eql/tests/encryption/tests/json_tree.rs b/packages/eql/tests/encryption/tests/json_tree.rs index 21c9fd05e..2903bb005 100644 --- a/packages/eql/tests/encryption/tests/json_tree.rs +++ b/packages/eql/tests/encryption/tests/json_tree.rs @@ -13,11 +13,12 @@ mod common; use eql_bindings::v4::json_tree::{ - JsonTree, JsonTreeDocument, JsonTreeEntry, JsonTreeLeafQuery, JsonTreeQuery, JsonTreeSubtree, + JsonTree, JsonTreeDocument, JsonTreeEntry, JsonTreeExtracted, JsonTreeLeafQuery, JsonTreeQuery, + JsonTreeSubtree, }; use eql_bindings::Identifier; use serde_json::{json, Map, Value}; -use stack_encrypt::json::{JsonDocument, JsonOptions, JsonPath, JsonSubtreeKey}; +use stack_encrypt::json::{JsonDocument, JsonOptions, JsonPath, JsonSelector, JsonSubtreeKey}; use stack_encrypt::plan::{Laid, ValuePlan}; use stack_encrypt::target::{Json, Layout}; use stack_encrypt::{op, NonEmpty, Plan}; @@ -120,16 +121,49 @@ async fn a_document_round_trips_through_the_typed_target() { ); // An entry lifted from the stored document, with the header and its - // ordinal grafted as `->` grafts them, opens to its scalar. + // ordinal grafted as `->` grafts them, opens to its scalar through the + // selector that asked for it, and only through that one. + let keyset = cipher.default_keyset(); + let selector = |path: &str| { + let path = JsonPath::parse(path).unwrap(); + let keyset = keyset.clone(); + async move { keyset.json_selector(&path, column()).await.unwrap() } + }; + let hello = selector("$.hello").await; let mut entry = stored.sv[1].clone(); entry.h = Some(stored.h.clone()); entry.j = Some(1); entry.i = Some(stored.i.clone()); entry.v = Some(stored.v); - let scalar: Value = cipher.decrypt_as(entry, column()).await.unwrap(); + let scalar: Value = cipher + .decrypt_as( + JsonTreeExtracted::from_selector(&hello, entry.clone()), + column(), + ) + .await + .unwrap(); assert_eq!(scalar, json!("world-1")); + let asked = JsonTreeExtracted::from_selector(&hello, entry.clone()); + assert_eq!(asked.entry(), &entry); + assert_eq!(asked.selector(), &hello); + let number = selector("$.number").await; + let refused = cipher + .decrypt_as::(JsonTreeExtracted::from_selector(&number, entry), column()) + .await; + assert!( + matches!( + refused, + Err(stack_encrypt::Error::JsonWrongEntry { + expected: "the entry at the path asked for" + }) + ), + "another path's selector refuses the entry: {refused:?}" + ); let bare = stored.sv[1].clone(); - assert!(cipher.decrypt_as::(bare, column()).await.is_err()); + assert!(cipher + .decrypt_as::(JsonTreeExtracted::from_selector(&hello, bare), column()) + .await + .is_err()); } #[tokio::test] @@ -484,14 +518,14 @@ fn postgres_extracts_walks_and_orders_by_path() { ) .unwrap(); } - let selector = |path: &str| { + let selector_of = |path: &str| -> JsonSelector { wait( &runtime, keyset.json_selector(&JsonPath::parse(path).unwrap(), column()), ) .unwrap() - .to_string() }; + let selector = |path: &str| selector_of(path).to_string(); let subtree_key = |path: &str| { wait( &runtime, @@ -506,8 +540,13 @@ fn postgres_extracts_walks_and_orders_by_path() { let entry = |row: &postgres::Row| -> JsonTreeEntry { serde_json::from_str(&row.get::<_, String>(0)).unwrap() }; - let open = - |e: JsonTreeEntry| -> Value { wait(&runtime, cipher.decrypt_as(e, column())).unwrap() }; + let open = |asked: &JsonSelector, e: JsonTreeEntry| -> Value { + wait( + &runtime, + cipher.decrypt_as(JsonTreeExtracted::from_selector(asked, e), column()), + ) + .unwrap() + }; // `->` by selector opens to the leaf; a missing path is NULL. let hello = selector("$.hello"); @@ -517,7 +556,7 @@ fn postgres_extracts_walks_and_orders_by_path() { &[&hello], ) .unwrap(); - assert_eq!(open(entry(&row)), json!("world-2")); + assert_eq!(open(&selector_of("$.hello"), entry(&row)), json!("world-2")); let missing = db .query_one( "SELECT (doc -> $1::text)::jsonb::text FROM json_tree_rows WHERE id = 3", @@ -545,7 +584,7 @@ fn postgres_extracts_walks_and_orders_by_path() { ) .unwrap() .iter() - .map(|r| open(entry(r))) + .map(|r| open(&selector_of("$.tags[*]"), entry(r))) .collect(); assert_eq!(tags, vec![json!("x"), json!("y")]); @@ -1326,6 +1365,175 @@ fn postgres_a_position_reaches_array_elements_only() { ); } +/// Needs an EQL v4 installation. What `->` and the walks return is checked +/// by the client to be what it asked for: every entry of a row is genuine +/// and opens under its own selector and position, so an answer made of the +/// wrong entries would open unless the client refuses it. Real rows come +/// back from PostgreSQL here, from a database answering the wrong question +/// (another path's entry, the sealed root, another position's subtree) and +/// from stored rows tampered with (a parent tag rewritten so the walk takes +/// a top-level field for a member, and two entries swapped in `sv`). Each +/// is refused: the wrong entry as `JsonWrongEntry`, a moved entry as +/// `Aead`, since its ciphertext is bound to the position it was sealed at. +#[test] +#[ignore = "requires EQL_TEST_DATABASE_URL pointing to an EQL v4 database"] +fn postgres_hostile_answers_are_refused_by_the_client() { + use stack_encrypt::Error; + + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let (cipher, _) = wait(&runtime, common::cipher()); + let keyset = cipher.default_keyset(); + let mut db = connect(); + db.batch_execute( + "CREATE TEMP TABLE json_tree_hostile (id int PRIMARY KEY, doc public.eql_v4_json_tree NOT NULL)", + ) + .unwrap(); + for id in [1, 2] { + let stored = serde_json::to_string(&wait(&runtime, seal(&cipher, &fixture()))).unwrap(); + db.execute( + "INSERT INTO json_tree_hostile VALUES ($1, $2::text::jsonb::public.eql_v4_json_tree)", + &[&id, &stored], + ) + .unwrap(); + } + let selector_of = |path: &str| -> JsonSelector { + wait( + &runtime, + keyset.json_selector(&JsonPath::parse(path).unwrap(), column()), + ) + .unwrap() + }; + let subtree_key = |path: &str| -> JsonSubtreeKey { + wait( + &runtime, + keyset.json_subtree_key( + &JsonPath::parse(path).unwrap(), + &JsonOptions::default(), + column(), + ), + ) + .unwrap() + }; + // `doc -> selector` in row `id`. + let arrow = |db: &mut postgres::Client, id: i32, selector: &JsonSelector| -> JsonTreeEntry { + let row = db + .query_one( + "SELECT (doc -> $1::text)::jsonb::text FROM json_tree_hostile WHERE id = $2", + &[&selector.to_string(), &id], + ) + .unwrap(); + serde_json::from_str(&row.get::<_, String>(0)).unwrap() + }; + // The walk `key` keys in row `id`, walked out (budget 0). + let walked = |db: &mut postgres::Client, id: i32, key: &JsonSubtreeKey| -> Vec { + db.query( + "SELECT e::jsonb::text FROM json_tree_hostile, eql_v4.json_tree_subtree_at(doc, $1::text, $2::text[], 0) e WHERE id = $3", + &[&key.token(), &key.gates(), &id], + ) + .unwrap() + .iter() + .map(|r| serde_json::from_str(&r.get::<_, String>(0)).unwrap()) + .collect() + }; + let extracted = |asked: &JsonSelector, entry: JsonTreeEntry| -> Result { + wait( + &runtime, + cipher.decrypt_as(JsonTreeExtracted::from_selector(asked, entry), column()), + ) + }; + let opened = |key: &JsonSubtreeKey, rows: Vec| -> Result { + wait( + &runtime, + cipher.decrypt_as(JsonTreeSubtree::from_key(key, rows), column()), + ) + }; + let wrong = |result: Result, what: &str| { + assert!( + matches!(result, Err(Error::JsonWrongEntry { .. })), + "{what}: {result:?}" + ); + }; + + // `->` on `$.hello` answered with the row's `$.number`, and with the + // sealed root, which opens to the whole document. + let hello = selector_of("$.hello"); + assert_eq!( + extracted(&hello, arrow(&mut db, 1, &hello)).unwrap(), + json!("world-1") + ); + wrong( + extracted(&hello, arrow(&mut db, 1, &selector_of("$.number"))), + "another path's entry", + ); + let root = arrow(&mut db, 1, &selector_of("$")); + wrong(extracted(&hello, root.clone()), "the sealed root"); + assert_eq!( + extracted(&selector_of("$"), root).unwrap(), + fixture(), + "the root asked for at `$`" + ); + + // The walk to `$.nested.list[1]` answered with the rows of the walk to + // `$.nested.list[0]`: the same selectors, another position. + let second = subtree_key("$.nested.list[1]"); + assert_eq!( + opened(&second, walked(&mut db, 1, &second)).unwrap(), + json!({"x": 2}) + ); + wrong( + opened( + &second, + walked(&mut db, 1, &subtree_key("$.nested.list[0]")), + ), + "another position's subtree", + ); + + // A stored row whose `$.number` (entry 9) carries a parent tag under + // `$.nested`'s token, at the ordinal after its two members: the walk + // takes it for nested's third member, and returns it right after + // nested's subtree, where it sits in `sv`. + let nested = subtree_key("$.nested"); + assert!(nested.gates().is_empty()); + assert_eq!( + opened(&nested, walked(&mut db, 1, &nested)).unwrap(), + fixture()["nested"] + ); + db.execute( + "UPDATE json_tree_hostile SET doc = jsonb_set(doc::jsonb, '{sv,9,p}', + to_jsonb(eql_v4_internal.json_tree_parent_tag( + eql_v4_internal.json_tree_iv(doc::jsonb ->> 'h'), decode($1::text, 'base64'), 2)))::public.eql_v4_json_tree + WHERE id = 1", + &[&nested.token()], + ) + .unwrap(); + let rows = walked(&mut db, 1, &nested); + assert_eq!(rows.len(), 7, "the walk took number for a member"); + assert_eq!( + rows.last().map(|e| (e.j, e.d)), + Some((Some(9), Some(1))), + "number, one level below nested" + ); + wrong(opened(&nested, rows), "a top-level field as a member"); + + // A stored row whose `deep` (3) and `number` (9) swapped places: each + // entry is found where it now sits, and fails to open there. + db.execute( + "UPDATE json_tree_hostile SET doc = jsonb_set(jsonb_set(doc::jsonb, + '{sv,3}', doc::jsonb -> 'sv' -> 9), '{sv,9}', doc::jsonb -> 'sv' -> 3)::public.eql_v4_json_tree + WHERE id = 2", + &[], + ) + .unwrap(); + let deep = selector_of("$.nested.deep"); + let moved = extracted(&deep, arrow(&mut db, 2, &deep)); + assert!(matches!(moved, Err(Error::Aead)), "`->`: {moved:?}"); + let moved = opened(&nested, walked(&mut db, 2, &nested)); + assert!(matches!(moved, Err(Error::Aead)), "the walk: {moved:?}"); +} + /// Needs an EQL v4 installation. A stored document is untrusted input: its /// parent tags are not authenticated, so whoever holds a token can write a /// chain of them deeper than any sealed document. A child's token is a diff --git a/packages/eql/tests/encryption/tests/json_tree/generated.rs b/packages/eql/tests/encryption/tests/json_tree/generated.rs index 192e0d7e5..0fb88343b 100644 --- a/packages/eql/tests/encryption/tests/json_tree/generated.rs +++ b/packages/eql/tests/encryption/tests/json_tree/generated.rs @@ -790,15 +790,16 @@ mod given_generated_documents_in_postgres { run("->", |scenario, case| { scenario.store(&case.docs); let keyset = scenario.cipher.default_keyset(); - let selectors: Vec = paths + // Each `->` answer opens only under the selector it was asked + // for, so the derived selectors are kept beside their text. + let asked: Vec = paths .iter() .map(|steps| { let path = JsonPath::parse(&format!("$.{}", steps.join("."))).unwrap(); - wait(&scenario.runtime, keyset.json_selector(&path, column())) - .unwrap() - .to_string() + wait(&scenario.runtime, keyset.json_selector(&path, column())).unwrap() }) .collect(); + let selectors: Vec = asked.iter().map(ToString::to_string).collect(); let keys: Vec = paths .iter() .map(|steps| format!("{{{}}}", steps.join(","))) @@ -817,7 +818,8 @@ mod given_generated_documents_in_postgres { let mut held: Vec = Vec::new(); let mut at: Vec<(String, i32)> = Vec::new(); for row in rows { - let path = format!("$.{}", paths[row.get::<_, i64>(0) as usize - 1].join(".")); + let i = row.get::<_, i64>(0) as usize - 1; + let path = format!("$.{}", paths[i].join(".")); let id: i32 = row.get(1); let want: Option = row.get(2); let kind: Option = row.get(3); @@ -837,7 +839,10 @@ mod given_generated_documents_in_postgres { let entry: JsonTreeEntry = serde_json::from_str(&got).unwrap(); let value = wait( &scenario.runtime, - scenario.cipher.decrypt_as::(entry, column()), + scenario.cipher.decrypt_as::( + JsonTreeExtracted::from_selector(&asked[i], entry), + column(), + ), ) .map_err(|e| { TestCaseError::fail(format!( diff --git a/packages/stack-encrypt/CHANGELOG.md b/packages/stack-encrypt/CHANGELOG.md index f2ce6411d..409199918 100644 --- a/packages/stack-encrypt/CHANGELOG.md +++ b/packages/stack-encrypt/CHANGELOG.md @@ -21,8 +21,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `EncryptFrom` / `DecryptInto` target, and it answers `op::Contains`, `op::ContainedBy`, `op::AtPath(op::Eq)` (a scalar at a path) and the orderings at an ordered path; `KeysetCipher::json_selector` - and `json_subtree_key` derive the extraction keys, and `JsonNode` and - `JsonSubtree` open what the server returns. `JsonOptions::sealed_root`, + and `json_subtree_key` derive the extraction keys. What the server + returns (`JsonNode` rows) opens only through what the client derived for + the read, which checks that the rows are the ones it asked for: an + entry's ciphertext is bound to its own selector and position, so any + entry of the row would otherwise open. `JsonSelector::open(node)` builds + a `JsonExtracted`, which refuses an entry that does not carry the + selector (another path's, or the sealed root in place of a node) before + any key is retrieved; `JsonSubtreeKey::open(rows)` builds a + `JsonSubtree`, which refuses a row that is not where the walk's rows must + be: a reached node without the path's selector or position, a row + without the selector of its parent's path and its own label (the key + holds the keyset's index-key PRF and the column's context to derive + it), and a row that does not follow the one before it in the document + or whose parent is not among the rows. Each refusal is the new + `Error::JsonWrongEntry`. Not checked, by design, is which row of the + column an entry comes from (ADR 0010). `JsonOptions::sealed_root`, on by default, makes the root entry the whole document so no token for the root exists. A document is at most 127 levels deep. New `TermError` variants: `JsonPath`, `JsonTooDeep`, `JsonTooWide`, `JsonSealedRoot`, diff --git a/packages/stack-encrypt/src/cipher.rs b/packages/stack-encrypt/src/cipher.rs index 62c79fa50..327da2785 100644 --- a/packages/stack-encrypt/src/cipher.rs +++ b/packages/stack-encrypt/src/cipher.rs @@ -173,6 +173,35 @@ pub enum Error { help("Open the value as the type it was sealed as. The ciphertext is authentic: it was sealed as another type, not altered.") )] OpenedType { expected: &'static str }, + /// The database answered a JSON index read with an entry other than + /// the one the read asked for: `->` returned the entry of another path, + /// or the sealed root in place of a node, or a subtree walk returned a + /// row that is not where the walk's key says its rows are (a node of + /// another path, a row out of its place in the document, a row whose + /// parent is not among the rows, or rows that stop before a node's + /// subtree is complete). Refused by the client, which + /// derived the path's selector itself; see + /// [`JsonSelector::open`](crate::json::JsonSelector::open) and + /// [`JsonSubtreeKey::open`](crate::json::JsonSubtreeKey::open). + /// + /// Not [`Aead`](Self::Aead): every entry may be genuine, sealed in + /// this column and this row. Not [`OpenedType`](Self::OpenedType) + /// either: the entry is not of an unexpected shape, it is the answer to + /// another question. A database that answers the query it was sent + /// never returns one, so this says the rows were tampered with or the + /// query that produced them was not the one the key or selector keyed. + /// + /// Which row a value belongs to is not checked: an entry from another + /// row of the same column at the same path, returned with that row's + /// header, opens (ADR 0010, `docs/adr/0010-what-a-stored-value-authenticates.md`). + /// + /// `expected` names what the read expected, never what the entry held. + #[error("the database returned a JSON index entry other than {expected}")] + #[diagnostic( + code(stack_encrypt::json_wrong_entry), + help("The entry opened, but it is not the one the query asked for. A database that answers the query it was sent never returns one: check that the rows came from the query the selector or subtree key was derived for, and treat them as tampered with if they did.") + )] + JsonWrongEntry { expected: &'static str }, /// ZeroKMS returned a different number of keys than were requested. #[error("expected {expected} data keys from ZeroKMS but received {received}")] #[diagnostic(code(stack_encrypt::key_count_mismatch))] @@ -343,7 +372,9 @@ impl crate::ErrorPayload for Error { ("expected", (*expected).into()), ("received", (*received).into()), ]), - Self::OpenedType { expected } => payload([("expected", (*expected).into())]), + Self::OpenedType { expected } | Self::JsonWrongEntry { expected } => { + payload([("expected", (*expected).into())]) + } Self::DescriptorTooLong { len } => payload([ ("len", (*len).into()), ("limit", Descriptor::MAX_LEN.into()), diff --git a/packages/stack-encrypt/src/codes.rs b/packages/stack-encrypt/src/codes.rs index f731ed26c..0d3836fc9 100644 --- a/packages/stack-encrypt/src/codes.rs +++ b/packages/stack-encrypt/src/codes.rs @@ -49,6 +49,9 @@ mod tests { Error::Kms(_) => Error::Kms(crate::kms::Error::Unexpected("kms".into())), Error::Aead => Error::Aead, Error::OpenedType { .. } => Error::OpenedType { expected: "i32" }, + Error::JsonWrongEntry { .. } => Error::JsonWrongEntry { + expected: "the entry at the path asked for", + }, Error::KeyCountMismatch { .. } => Error::KeyCountMismatch { expected: 2, received: 1, @@ -397,6 +400,12 @@ mod tests { Box::new(Error::OpenedType { expected: "i32" }), json!({ "expected": "i32" }), ), + ( + Box::new(Error::JsonWrongEntry { + expected: "the entry at the path asked for", + }), + json!({ "expected": "the entry at the path asked for" }), + ), ( Box::new(LeafBytesError::UnknownVersion(9)), json!({ "version": 9 }), diff --git a/packages/stack-encrypt/src/json/derive.rs b/packages/stack-encrypt/src/json/derive.rs index 17900588f..d50ab1f5f 100644 --- a/packages/stack-encrypt/src/json/derive.rs +++ b/packages/stack-encrypt/src/json/derive.rs @@ -8,6 +8,7 @@ //! ordinal `j` is its place in that order, and the AAD of its ciphertext //! binds it there. use std::collections::BTreeMap; +use std::sync::Arc; use serde_json::{Map, Number, Value}; use sha2::{Digest, Sha256}; @@ -16,15 +17,16 @@ use vitaminc_aead::{Cipher, Context, IntoAad, IntoContext, LocalCipherText}; use vitaminc_encrypt::{Aes256Cipher, Key as AesKey}; use vitaminc_hmac::HmacSha256Prf; use vitaminc_prf::{BlockVisitor, IntoPrfContext, PrfValue}; -use vitaminc_protected::{Controlled, Protected}; +use vitaminc_protected::{AsProtectedRef, Controlled, Protected}; -use super::path::{JsonPath, Segment}; +use super::path::{key_step_len, push_key, JsonPath, Segment, ARRAY_STEP}; use super::plain::{ insert_member, protected_array, protected_object, push_element, read_protected, take_protected, PlainJson, }; use super::query::{Gate, JsonSelector, JsonSubtreeKey}; -use super::{JsonDocument, JsonEntry, JsonHeader, JsonNode, JsonOptions, JsonSubtree, Tag}; +use super::{JsonDocument, JsonEntry, JsonExtracted, JsonHeader, JsonOptions, JsonSubtree, Tag}; +use crate::keyset::KeysetState; use crate::sem::TermError; use crate::target::{AeadContext, CallerContext, Decryption, Pending, Request}; use crate::{DecryptInto, Descriptor, Error, KeysetCipher, NonEmpty, StackCipher}; @@ -268,6 +270,18 @@ fn prf16( domain: &[u8], context: &Context<'_>, input: &[u8], +) -> Result { + prf16_of(prf, domain, context, input) +} + +/// [`prf16`] over any PRF input: a `Protected>` derives exactly +/// what its bytes do, so input built from decrypted plaintext stays +/// wrapped up to the PRF. +fn prf16_of( + prf: &HmacSha256Prf, + domain: &[u8], + context: &Context<'_>, + input: impl PrfValue, ) -> Result { let block: [u8; 32] = input .prf_visit_with_context( @@ -280,6 +294,53 @@ fn prf16( Ok(truncate(&block)) } +/// What derives path selectors under one column after the keyset handle +/// that derived a subtree key is gone: the keyset's loaded state (its +/// index-key PRF), shared, and the column's PRF context. A +/// [`JsonSubtreeKey`] holds one, so that opening a walk's rows can derive +/// the selector each row must carry from its parent's path and its own +/// decrypted label. Never printed; compared by keyset and context. +#[derive(Clone)] +pub(crate) struct Selectors { + keyset: Arc, + context: Context<'static>, +} + +impl Selectors { + pub(crate) fn new(keyset: Arc, context: Context<'static>) -> Self { + Self { keyset, context } + } + + /// The selector of the path whose selector bytes are `bytes`. + fn of(&self, bytes: Protected>) -> Result { + prf16_of(&self.keyset.prf, SELECTOR_DOMAIN, &self.context, bytes).map_err(Error::from) + } +} + +#[cfg(test)] +impl Selectors { + /// Selectors under a fixed test keyset (index key `[7; 32]`, as the + /// unit tests' `prf()`) and the column `users/profile`. + pub(crate) fn for_tests() -> Self { + use vitaminc_prf::PrfKeyInit; + Self::new( + Arc::new(KeysetState { + id: Uuid::from_u128(1), + name: None, + prf: HmacSha256Prf::new(Protected::new([7; 32])), + }), + Context::from_encoded(b"users/profile").into_owned(), + ) + } +} + +impl PartialEq for Selectors { + fn eq(&self, other: &Self) -> bool { + self.keyset.id == other.keyset.id && self.context == other.context + } +} +impl Eq for Selectors {} + /// The first sixteen bytes of SHA-256 over the concatenation of `parts`. /// Plain SHA-256 by design: the design's "Why plain SHA-256 is enough here" /// records why length extension does not apply. @@ -1076,22 +1137,35 @@ impl DecryptInto for JsonDocument { } } -/// Open one extracted node: the scalar it holds, or the whole document for -/// a sealed root. A container node holds no value of its own and is -/// refused with [`Error::OpenedType`]; its subtree opens through -/// [`JsonSubtree`]. -impl DecryptInto for JsonNode { +/// The refusal of a row that is not the one a read asked for. +fn wrong(expected: &'static str) -> Error { + Error::JsonWrongEntry { expected } +} + +/// Open the entry `->` returned for a selector: the scalar it holds, or the +/// whole document for a sealed root asked for at `$`. An entry that does +/// not carry the selector asked for is refused with +/// [`Error::JsonWrongEntry`] before any key is retrieved: it is another +/// path's entry, or the sealed root (whose selector is the root's) in place +/// of a node, and it would open, since every entry is sealed under its own +/// selector. A container node holds no value of its own and is refused +/// with [`Error::OpenedType`]; its subtree opens through [`JsonSubtree`]. +impl DecryptInto for JsonExtracted { type Context = AeadContext; fn decryption(self, context: Self::Context) -> Decryption { + let (node, selector) = self.into_parts(); + if node.entry.s != selector { + return Decryption::failed(wrong("the entry at the path asked for")); + } Decryption::from_open(move |cipher| { - let (request, aad) = match retrieval(&self.header, context) { + let (request, aad) = match retrieval(&node.header, context) { Ok(parts) => parts, Err(error) => return Pending::failed(cipher, error), }; Pending::request(cipher, vec![request], move |responses| { let key = responses.next_retrieved_key()?; let aes = leaf_cipher(&key)?; - let opened = open_entry(&aes, &self.header, &aad, &self.entry, self.j)?; + let opened = open_entry(&aes, &node.header, &aad, &node.entry, node.j)?; opened .whole .or(opened.scalar) @@ -1113,13 +1187,15 @@ impl DecryptInto for JsonNode { /// (`json_tree_subtree_at`); the children of the one node at the path, for /// a key without (`json_tree_subtree`). An object's members come in an /// order neither form fixes, and without their keys; open it as a -/// [`Value`] to keep them. +/// [`Value`] to keep them. Every row is checked against the key first +/// ([`JsonSubtreeKey::open`]); a row that does not fit is +/// [`Error::JsonWrongEntry`]. impl DecryptInto> for JsonSubtree { type Context = AeadContext; fn decryption(self, context: Self::Context) -> Decryption, K> { Decryption::from_open(move |cipher| { if self.is_sealed_root() { - let below = self.below; + let below = self.below(); return open_fallback(self, cipher, context).try_map(move |(path, mut whole)| { // The selected nodes move out of the document's // `Protected` into one of their own, leaving `null` @@ -1163,7 +1239,9 @@ impl DecryptInto> for JsonSubtree { /// descendants, and `->` on the node's selector returns the node itself. /// The sealed root a walk fell back to opens whole, and the key's path /// ([`JsonSubtreeKey::open`]) selects the one value under it; a path that -/// reaches no node, or more than one, is [`Error::OpenedType`]. +/// reaches no node, or more than one, is [`Error::OpenedType`]. Every row +/// is checked against the key first ([`JsonSubtreeKey::open`]); a row that +/// does not fit is [`Error::JsonWrongEntry`]. impl DecryptInto for JsonSubtree { type Context = AeadContext; fn decryption(self, context: Self::Context) -> Decryption { @@ -1191,9 +1269,9 @@ impl DecryptInto for JsonSubtree { } }); } - let reached = self.nodes.iter().any(|n| n.depth == Some(0)); + let below = self.below(); open_subtree(self, cipher, context).try_map(move |forest| { - if reached { + if !below { let mut values: Vec> = forest.into_iter().map(|(_, v)| v).collect(); return match values.len() { @@ -1230,14 +1308,27 @@ impl DecryptInto for JsonSubtree { } } +/// One row of a walk, opened: the node, and the selector, position and +/// depth the server returned it with. The depth is the server's report and +/// optional on the wire; the rows are placed by their positions and the +/// opened nodes' child counts, and a depth that disagrees with the place +/// a row is put in is refused. +struct Row { + node: Opened, + s: Tag, + j: u64, + d: Option, +} + /// Open every row and rebuild each complete subtree, labelled, each still -/// under its `Protected`. +/// under its `Protected`, checking every row against the key as it goes +/// ([`place`]). fn open_subtree<'a, K>( subtree: JsonSubtree, cipher: &'a StackCipher, context: AeadContext, ) -> Pending<'a, Vec<(Label, Protected)>, K> { - let mut nodes = subtree.nodes; + let (mut nodes, key) = subtree.into_parts(); nodes.sort_by_key(|n| n.j); // A walk returns each entry once. Two rows at one ordinal are the // server repeating an entry, which would let it line up one container @@ -1257,23 +1348,216 @@ fn open_subtree<'a, K>( Err(error) => return Pending::failed(cipher, error), }; Pending::request(cipher, vec![request], move |responses| { - let key = responses.next_retrieved_key()?; - let aes = leaf_cipher(&key)?; - let opened = nodes + let key_bytes = responses.next_retrieved_key()?; + let aes = leaf_cipher(&key_bytes)?; + let rows = nodes .iter() - .map(|n| open_entry(&aes, &header, &aad, &n.entry, n.j)) - .collect::, _>>()?; - let mut forest = Vec::new(); - let mut rest = opened.into_iter(); - while let Some(top) = rest.next() { - let (label, top) = take_label(top); - let value = rebuild(top, &mut rest, 0)?; - forest.push((label, value)); - } - Ok(forest) + .map(|n| { + Ok(Row { + node: open_entry(&aes, &header, &aad, &n.entry, n.j)?, + s: n.entry.s, + j: n.j, + d: n.depth, + }) + }) + .collect::, Error>>()?; + place(rows, &key) }) } +/// Put the opened rows of a walk back into the subtrees they hold, in +/// document order, refusing any row that is not where the key says the +/// walk's rows are ([`Error::JsonWrongEntry`]). +/// +/// Each entry opened under its own selector `s` and position `j`, which its +/// ciphertext is bound to, so each is a genuine node of the row; what the +/// check adds is that it is the node the walk asked for. A subtree's rows +/// in a document are its root and then every node beneath it, at +/// consecutive positions, each node's children in order; so the rows are +/// placed one by one, each the next position after the one before, one +/// level below its parent, and carrying the selector of its parent's path +/// extended by its own label, which the key derives from the decrypted +/// label. With gates, each subtree's root is a node the path reaches: the +/// path's selector, at depth `0`, at the position the path names if its +/// last step is one. Without gates, the rows are the children of the +/// path's node and their subtrees, side by side: each child carries the +/// selector of the path extended by its label, at depth `1`, right after +/// the subtree before it, and an element at the position after the +/// elements before it, from `0`. +fn place( + rows: Vec, + key: &JsonSubtreeKey, +) -> Result)>, Error> { + let below = key.gates.is_empty(); + let at = Protected::new(key.path.selector_bytes()); + let reached = key.selectors.of(at.clone())?; + let mut rest = rows.into_iter(); + let mut forest = Vec::new(); + // Without gates, the position the next child of the path's node must + // have: right after the subtree of the one before. + let mut next: Option = None; + while let Some(top) = rest.next() { + let (bytes, depth) = if below { + let bytes = child_selector_bytes(&at, &top.node.label) + .ok_or_else(|| wrong("a child of the walked node"))?; + // The walk does not return the node it starts from, so no row + // bounds where the children begin; an element's position does. + // The children of an array are its elements from `0`, so the + // `n`th row at depth 1 must be element `n`: a database that + // leaves out the first element would otherwise move every + // element down a position. A member's place is a keyed rank + // the client does not hold, so a member left out, and the last + // children of either, are still the database's word. + let out_of_place = + matches!(top.node.label, Label::Index(i) if i != forest.len() as u64); + if top.d.is_some_and(|d| d != 1) + || next.is_some_and(|j| j != top.j) + || out_of_place + || key.selectors.of(bytes.clone())? != top.s + { + return Err(wrong("a child of the walked node")); + } + (bytes, 1) + } else { + if top.d.is_some_and(|d| d != 0) + || top.s != reached + || !reaches(&key.path, &top.node.label) + { + return Err(wrong("a node the walked path reaches")); + } + (at.clone(), 0) + }; + let mut cursor = top + .j + .checked_add(1) + .ok_or_else(|| wrong("a node of the walk"))?; + let (label, node) = take_label(top.node); + let value = rebuild_walked(node, &bytes, depth, &mut rest, &mut cursor, &key.selectors)?; + next = Some(cursor); + forest.push((label, value)); + } + Ok(forest) +} + +/// Whether a node with `label` is one `path` reaches by its last step, +/// beyond what the selector says: a position names the element at that +/// position, where the selector reads every position as the wildcard. +fn reaches(path: &JsonPath, label: &Label) -> bool { + match path.segments().last() { + Some(Segment::Index(n)) => matches!(label, Label::Index(at) if at == n), + Some(Segment::Wildcard) => matches!(label, Label::Index(_)), + Some(Segment::Key(_)) => matches!(label, Label::Key(_)), + None => matches!(label, Label::Root), + } +} + +/// The selector bytes of the child labelled `label` of the node whose +/// selector bytes are `parent`: the parent's, then the child's step, a key +/// for a member and the array step for an element, as +/// [`JsonPath::selector_bytes`] writes them. A member's key is decrypted +/// plaintext, so the bytes are built inside a `Protected`, sized before +/// they are written. `None` for the root, which is no node's child. +fn child_selector_bytes(parent: &Protected>, label: &Label) -> Option>> { + let parent_len = parent.risky_ref().len(); + let mut out = match label { + Label::Root => return None, + Label::Index(_) => Protected::new(Vec::with_capacity(parent_len + 1)), + Label::Key(key) => Protected::new(Vec::with_capacity( + parent_len + key_step_len(key.risky_ref().len()), + )), + }; + out.update_with_ref(parent.as_protected_ref(), |out, parent: &Vec| { + out.extend_from_slice(parent); + }); + match label { + Label::Key(key) => out.update_with_ref(key.as_protected_ref(), |out, key: &String| { + push_key(out, key); + }), + _ => out.update(|out| out.push(ARRAY_STEP)), + } + Some(out) +} + +/// [`rebuild`] for a walk's rows, checking each child as it is taken: +/// that it is at `cursor`, the position after the row before it; that it +/// is one level below `node`, which sits `d` levels below the walk's start; +/// and that it carries the selector of `node`'s path, whose selector bytes +/// are `bytes`, extended by its own label. A sealed root among the rows is +/// not a node of any walk. Rows that end before `node`'s authenticated +/// child count is met are refused like a gap: the walk returns a node's +/// whole subtree, so the database left the next node out. +/// +/// The rows are consecutive entries of one sealed document, placed by its +/// authenticated child counts, so they nest no deeper than the document +/// does, which is never deeper than [`MAX_DEPTH`]; a document that does is +/// refused as a shape mismatch rather than recursed into. +fn rebuild_walked( + node: Opened, + bytes: &Protected>, + d: u32, + rest: &mut impl Iterator, + cursor: &mut u64, + selectors: &Selectors, +) -> Result, Error> { + if node.whole.is_some() { + return Err(wrong("a node of the walked subtree")); + } + if d as usize > MAX_DEPTH { + return Err(shape("a JSON document no deeper than the index seals")); + } + let child = |rest: &mut dyn Iterator, cursor: &mut u64| { + // The walk returns a node's whole subtree and stops at no budget + // short of it (the SQL falls back to the sealed root instead), so + // rows that run out before a child count is met are a row left out + // at the end, the same refusal as one left out in the middle. + let row = rest + .next() + .ok_or_else(|| wrong("the next node of the walked subtree"))?; + let child_bytes = child_selector_bytes(bytes, &row.node.label) + .ok_or_else(|| wrong("the next node of the walked subtree"))?; + if row.j != *cursor + || row.d.is_some_and(|got| Some(got) != d.checked_add(1)) + || selectors.of(child_bytes.clone())? != row.s + { + return Err(wrong("the next node of the walked subtree")); + } + *cursor = cursor + .checked_add(1) + .ok_or_else(|| wrong("the next node of the walked subtree"))?; + Ok::<_, Error>((row.node, child_bytes)) + }; + match node.kind { + Kind::Object => { + let mut object = protected_object(); + for _ in 0..node.children { + let (member, member_bytes) = child(rest, cursor)?; + let (label, member) = take_label(member); + let Label::Key(key) = label else { + return Err(shape("an object member")); + }; + let value = rebuild_walked(member, &member_bytes, d + 1, rest, cursor, selectors)?; + insert_member(&mut object, key, value); + } + Ok(object) + } + Kind::Array => { + let mut array = protected_array(node.children as usize); + for i in 0..node.children { + let (element, element_bytes) = child(rest, cursor)?; + if !matches!(element.label, Label::Index(at) if at == u64::from(i)) { + return Err(shape("an array element in order")); + } + push_element( + &mut array, + rebuild_walked(element, &element_bytes, d + 1, rest, cursor, selectors)?, + ); + } + Ok(array) + } + _ => node.scalar.ok_or_else(|| shape("a JSON scalar")), + } +} + /// The values of a node's children, in the node's order, moved out of it: /// an array's elements, an object's member values (each leaving `null` /// under its key, which stays in the document to be wiped with it), and @@ -1332,11 +1616,12 @@ fn open_fallback<'a, K>( /// position with the element selector, so a member that happens to sit at /// that ordinal under an object is not taken for an element. fn subtree_key_for( - prf: &HmacSha256Prf, - context: &Context<'_>, + selectors: Selectors, options: &JsonOptions, path: &JsonPath, ) -> Result { + let prf = &selectors.keyset.prf; + let context = &selectors.context; let segments = path.segments(); let (head, rest) = if options.sealed_root { let Some((first, rest)) = segments.split_first() else { @@ -1391,6 +1676,7 @@ fn subtree_key_for( token, gates, path: path.clone(), + selectors, }) } @@ -1437,7 +1723,8 @@ impl KeysetCipher<'_, K> { descriptor: NonEmpty>, ) -> Pending<'_, JsonSubtreeKey, K> { let context = descriptor.into_prf_context().into_owned(); - let key = subtree_key_for(self.prf(), &context, options, path); + let selectors = Selectors::new(Arc::clone(self.state()), context); + let key = subtree_key_for(selectors, options, path); Pending::ready(self, key.map_err(Error::from)) } } @@ -1545,10 +1832,13 @@ mod tests { #[test] fn the_subtree_key_derives_what_it_can_and_gates_the_rest() { - let context = Context::from_encoded(b"users/profile"); let value = json!({"a": {"b": [1, {"c": 2}]}, "l": [[0], 1]}); let key = |options: &JsonOptions, path: &str| { - subtree_key_for(&prf(), &context, options, &JsonPath::parse(path).unwrap()) + subtree_key_for( + Selectors::for_tests(), + options, + &JsonPath::parse(path).unwrap(), + ) }; let at = |t: &JsonTerms, path: &str| { t.nodes() @@ -1617,11 +1907,9 @@ mod tests { #[test] fn a_position_past_postgres_integer_is_refused_not_sent() { - let context = Context::from_encoded(b"users/profile"); let key = |path: &str| { subtree_key_for( - &prf(), - &context, + Selectors::for_tests(), &JsonOptions::default(), &JsonPath::parse(path).unwrap(), ) @@ -1843,6 +2131,92 @@ mod tests { assert!(matches!(sealed(longest + 1), Err(Error::Other(_)))); } + mod given_a_walked_row { + use super::*; + + fn path(text: &str) -> JsonPath { + JsonPath::parse(text).unwrap() + } + + fn key(text: &str) -> Label { + Label::Key(Protected::new(text.to_owned())) + } + + #[test] + fn a_childs_selector_bytes_are_its_paths() { + let parent = Protected::new(path("$.a[3]").selector_bytes()); + for (label, child) in [(key("b"), "$.a[*].b"), (Label::Index(7), "$.a[*][*]")] { + let bytes = child_selector_bytes(&parent, &label).expect("a child"); + assert_eq!( + bytes.risky_ref(), + &path(child).selector_bytes(), + "{child}: the parent's bytes and the child's step" + ); + assert_eq!( + bytes.risky_ref().len(), + bytes.risky_ref().capacity(), + "{child}: sized before it is written" + ); + } + assert!( + child_selector_bytes(&parent, &Label::Root).is_none(), + "the root is no node's child" + ); + } + + #[test] + fn the_last_step_of_a_path_names_the_label_it_reaches() { + assert!(reaches(&path("$.a[2]"), &Label::Index(2)), "the position"); + assert!(!reaches(&path("$.a[2]"), &Label::Index(1)), "another one"); + assert!(!reaches(&path("$.a[2]"), &key("2")), "a member"); + assert!(reaches(&path("$.a[*]"), &Label::Index(9)), "any element"); + assert!(!reaches(&path("$.a[*]"), &key("x")), "not a member"); + assert!(reaches(&path("$.a.b"), &key("b")), "a member"); + assert!(!reaches(&path("$.a.b"), &Label::Index(0)), "not an element"); + assert!(reaches(&path("$"), &Label::Root), "the root"); + assert!(!reaches(&path("$"), &key("a")), "only the root"); + } + + #[test] + fn selectors_are_the_ones_the_key_derives_and_compare_by_keyset_and_column() { + let selectors = Selectors::for_tests(); + let at = |text: &str| { + prf16( + &prf(), + SELECTOR_DOMAIN, + &Context::from_encoded(b"users/profile"), + &path(text).selector_bytes(), + ) + .unwrap() + }; + assert_eq!( + selectors + .of(Protected::new(path("$.a.b").selector_bytes())) + .unwrap(), + at("$.a.b"), + "a protected input derives what its bytes do" + ); + assert!( + selectors == Selectors::for_tests(), + "same keyset, same column" + ); + let other_column = Selectors::new( + Arc::clone(&selectors.keyset), + Context::from_encoded(b"users/bio").into_owned(), + ); + assert!(selectors != other_column, "another column"); + let other_keyset = Selectors::new( + Arc::new(KeysetState { + id: Uuid::from_u128(2), + name: None, + prf: prf(), + }), + selectors.context.clone(), + ); + assert!(selectors != other_keyset, "another keyset"); + } + } + #[test] fn too_deep_a_document_is_refused() { // The innermost scalar of `n` nested arrays sits at depth `n`. diff --git a/packages/stack-encrypt/src/json/mod.rs b/packages/stack-encrypt/src/json/mod.rs index ff59c71f0..a4ee371c0 100644 --- a/packages/stack-encrypt/src/json/mod.rs +++ b/packages/stack-encrypt/src/json/mod.rs @@ -48,8 +48,13 @@ //! `AtPath`), [`JsonLeafQuery`] (ordering at a path), and the two //! extraction keys a caller derives through the keyset cipher, //! [`JsonSelector`] and [`JsonSubtreeKey`]. What the server returns for an -//! extraction opens through [`JsonNode`] (one entry) and [`JsonSubtree`] -//! (the entries under a token). +//! extraction is [`JsonNode`] rows, which open only through what the +//! client derived for the read: [`JsonSelector::open`] for the one entry +//! `->` returns ([`JsonExtracted`]), [`JsonSubtreeKey::open`] for the rows +//! of a walk ([`JsonSubtree`]). Each checks that the rows are the ones the +//! read asked for, which their ciphertexts alone do not say: an entry is +//! sealed under its own selector and position, so any entry of the row +//! opens. //! //! # Where this deviates from the design //! @@ -528,9 +533,13 @@ where /// One entry as an extraction returns it: the entry, the document's header /// grafted beside it, and the entry's ordinal `j` in the document, which -/// its ciphertext is bound to. What `->` and the walk return per row, and -/// what [`DecryptInto`](crate::DecryptInto) opens to the node's -/// scalar, or to the whole document for a sealed root. +/// its ciphertext is bound to. What `->` and the walk return per row. +/// +/// A row is the database's word, so it does not open on its own: the +/// selector the client derived opens the entry `->` returned +/// ([`JsonSelector::open`]), and the key that keyed a walk opens its rows +/// ([`JsonSubtreeKey::open`]), each after checking the rows are the ones +/// it asked for. /// /// Deserializes from the entry's wire fields plus `h` and `j`, and tolerates /// the envelope fields the SQL grafts beside them (`i`, `v`, `d`). @@ -562,10 +571,43 @@ impl JsonNode { } } -impl Decryptable for JsonNode { +/// The entry `->` returned, with the selector the client asked for: what +/// [`DecryptInto`](crate::DecryptInto) opens to the node's scalar, +/// or to the whole document for a sealed root asked for at `$`. Built only +/// by [`JsonSelector::open`], so it always carries the selector the entry +/// is checked against; an entry carrying another is refused with +/// [`Error::JsonWrongEntry`](crate::Error::JsonWrongEntry) before any key +/// is retrieved. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct JsonExtracted { + node: JsonNode, + selector: Tag, +} + +impl JsonExtracted { + pub(crate) fn keyed(node: JsonNode, selector: Tag) -> Self { + Self { node, selector } + } + + /// The row, as the database returned it. + pub fn node(&self) -> &JsonNode { + &self.node + } + + /// The selector the read asked for. + pub fn selector(&self) -> JsonSelector { + JsonSelector(self.selector) + } + + pub(crate) fn into_parts(self) -> (JsonNode, Tag) { + (self.node, self.selector) + } +} + +impl Decryptable for JsonExtracted { const DECRYPTABLE: bool = true; } -impl DecryptField for JsonNode +impl DecryptField for JsonExtracted where Self: crate::DecryptInto

, Ctx: Into<>::Context>, @@ -580,7 +622,9 @@ where /// token, gates)` yields, collected, and what /// [`DecryptInto`](crate::DecryptInto) reassembles into the value /// under the walked path. Built only by the key that keyed the walk -/// ([`JsonSubtreeKey::open`]), so it always carries the walked path. +/// ([`JsonSubtreeKey::open`]), so it always carries the walked path, and +/// what checks every row against it when the rows are opened (see +/// [`JsonSubtreeKey::open`] for the checks). /// /// The ungated walk returns the node's descendants and not the node /// itself, so the node's kind is read from its children's labels: elements @@ -599,15 +643,15 @@ where #[derive(Clone, Debug, PartialEq, Eq)] pub struct JsonSubtree { nodes: Vec, - path: JsonPath, - below: bool, + key: JsonSubtreeKey, } impl JsonSubtree { - /// The rows of a walk keyed from `path`: below the path's node when - /// `below` (`json_tree_subtree`), at the nodes it reaches otherwise. - pub(crate) fn keyed(nodes: Vec, path: JsonPath, below: bool) -> Self { - Self { nodes, path, below } + /// The rows of a walk `key` keyed: below the path's node when the key + /// has no gates (`json_tree_subtree`), at the nodes it reaches + /// otherwise. + pub(crate) fn keyed(nodes: Vec, key: JsonSubtreeKey) -> Self { + Self { nodes, key } } /// The rows. @@ -617,7 +661,18 @@ impl JsonSubtree { /// The path the walk was keyed from. pub fn path(&self) -> &JsonPath { - &self.path + &self.key.path + } + + /// Whether the rows are the descendants of the path's node + /// (`json_tree_subtree`, a key with no gates) rather than the nodes + /// the path reaches. + pub(crate) fn below(&self) -> bool { + self.key.gates.is_empty() + } + + pub(crate) fn into_parts(self) -> (Vec, JsonSubtreeKey) { + (self.nodes, self.key) } /// Whether the rows are the sealed root the server returned in place diff --git a/packages/stack-encrypt/src/json/path.rs b/packages/stack-encrypt/src/json/path.rs index 2545defe1..77f86a457 100644 --- a/packages/stack-encrypt/src/json/path.rs +++ b/packages/stack-encrypt/src/json/path.rs @@ -62,7 +62,7 @@ impl crate::ErrorPayload for JsonPathError { const KEY_STEP: u8 = 0x01; /// The tag byte of an array step (positional or wildcard) in a path's /// canonical bytes. -const ARRAY_STEP: u8 = 0x02; +pub(crate) const ARRAY_STEP: u8 = 0x02; impl JsonPath { /// The root, `$`. @@ -308,13 +308,21 @@ impl JsonPath { } } -/// A key step: the tag, the key's length and the key. -fn push_key(out: &mut Vec, key: &str) { +/// A key step: the tag, the key's length and the key. What +/// [`JsonPath::selector_bytes`] writes for a key, and what a walk's rows +/// are checked against when the client extends a path by a member's +/// decrypted key. +pub(crate) fn push_key(out: &mut Vec, key: &str) { out.push(KEY_STEP); out.extend_from_slice(&(key.len() as u32).to_le_bytes()); out.extend_from_slice(key.as_bytes()); } +/// How many bytes [`push_key`] writes for a key of `len` bytes. +pub(crate) const fn key_step_len(len: usize) -> usize { + 1 + 4 + len +} + /// The JSON string at the front of `text`, which starts at its opening /// quote: its value and the text after its closing quote. The escapes are /// JSON's. The refusal is the reason; the caller names the offset. diff --git a/packages/stack-encrypt/src/json/query.rs b/packages/stack-encrypt/src/json/query.rs index e515723ce..8ca017004 100644 --- a/packages/stack-encrypt/src/json/query.rs +++ b/packages/stack-encrypt/src/json/query.rs @@ -12,9 +12,9 @@ use std::fmt; use serde::{Deserialize, Serialize}; use serde_json::Value; -use super::derive::{JsonTerms, TermNode}; +use super::derive::{JsonTerms, Selectors, TermNode}; use super::path::JsonPath; -use super::{base64, JsonDocument, JsonNode, JsonOptions, JsonSubtree, Tag}; +use super::{base64, JsonDocument, JsonExtracted, JsonNode, JsonOptions, JsonSubtree, Tag}; use crate::op::{ Answers, AtPath, Between, ContainedBy, Contains, DefaultIndex, Eq, Gt, Gte, Input, LabelledTarget, Lt, Lte, One, Pair, Pathed, QueryTerm, QueryToken, Supports, @@ -296,6 +296,10 @@ impl<'de> Deserialize<'de> for JsonLeafQuery { /// A path selector: what `->` and the path functions take. Prints and /// serializes as the base64 the stored entry carries in `s`. +/// +/// It is also what opens the entry `->` returns ([`open`](Self::open)): +/// the selector the client derived for the path is the one the entry must +/// carry, so a database that answers with another path's entry is refused. #[derive(Clone, Copy, PartialEq, Eq, Hash)] pub struct JsonSelector(pub(crate) Tag); @@ -304,6 +308,24 @@ impl JsonSelector { pub fn as_bytes(&self) -> &Tag { &self.0 } + + /// The entry `col -> selector` returned, ready to open: the only way to + /// open one. Opening it ([`DecryptInto`](crate::DecryptInto)) + /// first checks that the entry carries this selector, and refuses any + /// other with [`Error::JsonWrongEntry`] before a key is retrieved: an + /// entry's ciphertext is bound to the selector it carries, so a + /// database that answers `$.salary` with the row's `$.bonus`, or with + /// the sealed root (the whole document, whose selector is the root's), + /// returns an entry that would open, and is not the one asked for. + /// + /// What the check covers is the path, read as a selector reads it: a + /// position reads as the wildcard, so any element at the path passes, + /// as `->` returns the first one. It does not cover the row: the same + /// path's entry from another row of the column, returned with that + /// row's header, opens (ADR 0010). + pub fn open(&self, node: JsonNode) -> JsonExtracted { + JsonExtracted::keyed(node, self.0) + } } impl fmt::Display for JsonSelector { @@ -355,11 +377,18 @@ impl Gate { /// derived from, which [`open`](Self::open) attaches to the rows the walk /// returns. Serializes as `{"t": "…", "g": ["…", "#0:…", …]}`, `g` omitted /// when empty; the path is not part of the wire. +/// +/// The key also holds what derives selectors under the column it was +/// derived for (the keyset's index-key PRF, shared with the keyset handle, +/// and the column's context), so that opening the rows can check each +/// one's selector against the path it must sit at. Neither is printed or +/// serialized. #[derive(Clone, PartialEq, Eq)] pub struct JsonSubtreeKey { pub(crate) token: Tag, pub(crate) gates: Vec, pub(crate) path: JsonPath, + pub(crate) selectors: Selectors, } impl JsonSubtreeKey { @@ -397,8 +426,31 @@ impl JsonSubtreeKey { /// token, gates)` and are the nodes the path reaches, with theirs. The /// key knows which, so the fallback opens to what the walked rows /// open to either way. + /// + /// Opening the rows checks each one against the key, and refuses the + /// rows with [`Error::JsonWrongEntry`] if one does not fit: a node the + /// path reaches must carry the path's selector, at depth `0`, with the + /// position the path names if it ends in one; every other row must + /// carry the selector of its parent's path extended by its own label + /// (its key, or the wildcard for an array element), sit one level + /// below its parent, and follow the row before it in the document with + /// no entry between, so that each node's rows are exactly its subtree. + /// A row whose parent is not among the rows is refused. + /// + /// With no gates the walk does not return the node it starts from, so + /// its children are checked against each other: they follow one + /// another with no gap, and an array's start at element `0`. + /// + /// Not covered: which row of the column the entries come from (ADR + /// 0010), and, until the walk returns the node it starts from (#1356), + /// two things about a walk with no gates: which node at the path's + /// selector it descends from, and whether children were left out after + /// the last row, or members anywhere (a member's place is a keyed rank + /// the client does not hold). An array position before the last step + /// of a path reads as the wildcard to the selectors, so another + /// element's subtree at the same depth passes too. pub fn open(&self, rows: Vec) -> JsonSubtree { - JsonSubtree::keyed(rows, self.path.clone(), self.gates.is_empty()) + JsonSubtree::keyed(rows, self.clone()) } } @@ -782,6 +834,7 @@ mod tests { token: [0xcd; 16], gates: vec![Gate::Selector([0xab; 16]), Gate::Slot(3, [0xab; 16])], path: JsonPath::parse("$.a.b[3]").unwrap(), + selectors: Selectors::for_tests(), }; assert_eq!(key.token(), "zc3Nzc3Nzc3Nzc3Nzc3NzQ=="); assert_eq!(key.token_bytes(), &[0xcd; 16]); @@ -802,6 +855,7 @@ mod tests { token: [0xcd; 16], gates: Vec::new(), path: JsonPath::parse("$.a").unwrap(), + selectors: Selectors::for_tests(), }; assert_eq!( serde_json::to_value(&bare).unwrap(), diff --git a/packages/stack-encrypt/src/keyset.rs b/packages/stack-encrypt/src/keyset.rs index 78638ddc5..c1d891e5a 100644 --- a/packages/stack-encrypt/src/keyset.rs +++ b/packages/stack-encrypt/src/keyset.rs @@ -614,6 +614,13 @@ impl<'k, K> KeysetCipher<'k, K> { &self.state.prf } + /// The keyset's loaded state, shared: for a value that must derive + /// terms after this handle is gone, such as a JSON subtree key, which + /// derives the selectors a walk's rows must carry when it opens them. + pub(crate) fn state(&self) -> &Arc { + &self.state + } + /// The underlying data-key source. pub fn kms(&self) -> &'k K { self.cipher.kms() diff --git a/packages/stack-encrypt/tests/json_index.rs b/packages/stack-encrypt/tests/json_index.rs index f4d42856e..57059dc17 100644 --- a/packages/stack-encrypt/tests/json_index.rs +++ b/packages/stack-encrypt/tests/json_index.rs @@ -22,7 +22,8 @@ mod common; use common::stack_cipher; use serde_json::{json, Map, Value}; use stack_encrypt::json::{ - JsonDocument, JsonNode, JsonOptions, JsonPath, JsonQuery, JsonSubtree, JsonSubtreeKey, Tag, + JsonDocument, JsonExtracted, JsonNode, JsonOptions, JsonPath, JsonQuery, JsonSelector, + JsonSubtree, JsonSubtreeKey, Tag, }; use stack_encrypt::op; use stack_encrypt::sem::TermError; @@ -163,6 +164,24 @@ async fn subtree_key( .await } +/// The selector of `path` under the suite's column. +async fn selector_of(cipher: &Cipher, path: &str) -> JsonSelector { + cipher + .default_keyset() + .json_selector( + &JsonPath::parse(path).unwrap(), + nonempty!("users").with("profile"), + ) + .await + .unwrap() +} + +/// `node` as `->` on `path` returned it, opened by the selector derived +/// for `path`: the only way to open an extracted entry. +async fn extracted(cipher: &Cipher, path: &str, node: JsonNode) -> JsonExtracted { + selector_of(cipher, path).await.open(node) +} + /// `rows` as the key derived for `path` opens them: the only way to build /// a [`JsonSubtree`]. async fn keyed(cipher: &Cipher, path: &str, rows: Vec) -> JsonSubtree { @@ -926,24 +945,32 @@ mod given_an_extracted_node { let Scenario { cipher, .. } = scenario().await; let doc = seal(&cipher, &fixture()).await; let node = |j: usize| doc.node(j).unwrap(); - let scalar = |j: usize| cipher.decrypt_as::(node(j), aead()); + let scalar = |path: &'static str, j: usize| { + let cipher = &cipher; + let node = node(j); + async move { + cipher + .decrypt_as::(extracted(cipher, path, node).await, aead()) + .await + } + }; assert_eq!( - scalar(1).await.unwrap(), + scalar("$.hello", 1).await.unwrap(), json!("world-1"), "$.hello opens to its scalar" ); assert_eq!( - scalar(6).await.unwrap(), + scalar("$.nested.list[0].x", 6).await.unwrap(), json!(1), "[0].x opens to its scalar" ); assert_eq!( - scalar(0).await.unwrap(), + scalar("$", 0).await.unwrap(), fixture(), "the sealed root is the document" ); assertions::refused_as_opened_type( - &scalar(2).await, + &scalar("$.nested", 2).await, None, "a container node holds no value of its own", ); @@ -971,19 +998,20 @@ mod given_an_extracted_node { "as a list, the walk opens to $.nested's members" ); - // The list's token is the server's to derive, one level under nested. - let list = child_token(&doc, key.token_bytes(), 4); + // The list is one gate below nested: the walk reaches it and returns + // it with its subtree. + let list = subtree_key(&cipher, "$.nested.list", JsonOptions::default()) + .await + .unwrap(); + let rows = gated_walk(&doc, &list); let elements = cipher - .decrypt_as::( - keyed(&cipher, "$.nested.list", subtree_nodes(&doc, &list)).await, - aead(), - ) + .decrypt_as::(list.open(rows), aead()) .await .unwrap(); assert_eq!( elements, json!([{"x": 1}, {"x": 2}]), - "the walk under the list opens to the list" + "the walk to the list opens to the list" ); } @@ -1000,7 +1028,10 @@ mod given_an_extracted_node { assert_eq!(back.ordinal(), 6, "the node keeps its ordinal"); assert_eq!( cipher - .decrypt_as::(back, aead()) + .decrypt_as::( + extracted(&cipher, "$.nested.list[*].x", back).await, + aead() + ) .await .unwrap(), json!(1), @@ -1104,19 +1135,17 @@ mod given_gated_walk_rows { Some("one node at the walked path"), "two reached nodes are not one value", ); - // Without a d = 0 row the same entries are the children of one node and - // assemble into it. - let elements = cipher - .decrypt_as::( - keyed(&cipher, "$.nested.list", subtree_nodes(&doc, &list)).await, - aead(), - ) - .await - .unwrap(); - assert_eq!( - elements, - json!([{"x": 1}, {"x": 2}]), - "without a d = 0 row the rows assemble into their parent" + // The key says which walk the rows came from: the children of the + // list, which an ungated walk from the list's token would return, are + // not what the gated walk to the list returns, and are refused. + assertions::refused_as_wrong_entry( + &cipher + .decrypt_as::( + keyed(&cipher, "$.nested.list", subtree_nodes(&doc, &list)).await, + aead(), + ) + .await, + "the ungated walk's rows under the gated key", ); } } @@ -1131,19 +1160,22 @@ mod given_a_derived_record { async fn opens_each_stored_form_as_its_field() { let Scenario { cipher, .. } = scenario().await; let doc = seal(&cipher, &fixture()).await; - let node = doc.node(1).unwrap(); - let key = subtree_key(&cipher, "$.nested", JsonOptions::default()) + let node = extracted(&cipher, "$.hello", doc.node(1).unwrap()).await; + let list = subtree_key(&cipher, "$.nested.list", JsonOptions::default()) .await .unwrap(); - let list = child_token(&doc, key.token_bytes(), 4); - let rows = subtree_nodes(&doc, &list); - let subtree = keyed(&cipher, "$.nested.list", rows.clone()).await; + let rows = gated_walk(&doc, &list); + let subtree = list.open(rows.clone()); assert_eq!( subtree.nodes(), rows.as_slice(), "a subtree holds the rows it was opened with" ); - assert_eq!(subtree.nodes().len(), 4, "both elements and both leaves"); + assert_eq!( + subtree.nodes().len(), + 5, + "the list, both elements and both leaves" + ); let opening: Option> = doc.decryption_field(aead()); let opened = cipher @@ -1247,8 +1279,9 @@ mod given_a_fallback_root { } /// The same root at depth 1, or at depth 0 under an unsealed document, - /// is a walk's row and not the fallback; neither assembles. A fallback - /// with no path cannot be built: the key is the only constructor. + /// is a walk's row and not the fallback, and not a child of the walked + /// node either: its selector is the root's. A fallback with no path + /// cannot be built: the key is the only constructor. #[tokio::test] async fn is_only_the_sealed_root_alone_at_depth_zero() { let Scenario { cipher, .. } = scenario().await; @@ -1259,10 +1292,9 @@ mod given_a_fallback_root { !rows.is_sealed_root(), "the root at depth 1 is a walk's row" ); - assertions::refused_as_opened_type( + assertions::refused_as_wrong_entry( &cipher.decrypt_as::(rows, aead()).await, - None, - "the root at depth 1 does not assemble", + "the root at depth 1 is not a row of the walk", ); let open_doc = seal_with(&cipher, &fixture(), &open_root()).await; assert!( @@ -1396,9 +1428,8 @@ mod given_rows_the_server_chose { /// The rows of a walk come from the server, which can repeat an entry or /// pick which entries to return. Every entry is authentic, but the /// rebuild nests the containers it is handed: a repeated row is refused, - /// and so is a line of containers deeper than any document the index - /// seals, as a shape mismatch rather than a recursion as deep as the - /// rows. + /// and so is a line of containers that are not one subtree's consecutive + /// entries, before the rebuild can recurse as deep as the rows. #[tokio::test] async fn refuses_a_repeated_row_or_a_line_of_containers() { let Scenario { cipher, .. } = scenario().await; @@ -1428,14 +1459,13 @@ mod given_rows_the_server_chose { .map(|j| doc.node(j).unwrap()) .collect(); assert_eq!(containers.len(), 201, "200 objects and the last leaf"); - assertions::refused_as_opened_type( + assertions::refused_as_wrong_entry( &cipher .decrypt_as::, JsonSubtree>( keyed(&cipher, "$.k000", containers).await, aead(), ) .await, - None, "a line of 200 objects", ); @@ -1457,14 +1487,13 @@ mod given_rows_the_server_chose { 202, "the outer array, 200 inner and the last leaf" ); - assertions::refused_as_opened_type( + assertions::refused_as_wrong_entry( &cipher .decrypt_as::, JsonSubtree>( keyed(&cipher, "$.k000", arrays).await, aead(), ) .await, - None, "a line of 202 arrays", ); } @@ -1523,7 +1552,9 @@ mod given_a_tampered_document { lifted["j"] = json!(2); let lifted: JsonNode = serde_json::from_value(lifted).unwrap(); assertions::refused_as_tampered( - &cipher.decrypt_as::(lifted, aead()).await, + &cipher + .decrypt_as::(extracted(&cipher, "$.a", lifted).await, aead()) + .await, "a node under another ordinal", ); } @@ -2815,6 +2846,15 @@ mod assertions { ); } + /// `result` is the refusal of an entry that is genuine but is not the + /// one asked for: another path's, or a row the walk did not reach. + pub(super) fn refused_as_wrong_entry(result: &Result, what: &str) { + assert!( + matches!(result, Err(Error::JsonWrongEntry { .. })), + "{what}: should be refused as the wrong entry, got {result:?}" + ); + } + /// `result` is tamper evidence: the authentication failed. pub(super) fn refused_as_tampered(result: &Result, what: &str) { assert!( diff --git a/packages/stack-encrypt/tests/json_read_checks.rs b/packages/stack-encrypt/tests/json_read_checks.rs new file mode 100644 index 000000000..53f0f626a --- /dev/null +++ b/packages/stack-encrypt/tests/json_read_checks.rs @@ -0,0 +1,592 @@ +//! What the client checks when it opens the rows a JSON index read +//! returned, with no database: the entry `->` returns must carry the +//! selector the client derived for the path, and every row of a subtree +//! walk must sit where the key that keyed the walk says its rows are. +//! Each entry's ciphertext is bound to its own selector and position, so +//! without these checks any entry of the row would open in place of the one +//! asked for. +//! +//! The rows here are built as the SQL returns them (the entry, the header +//! and the ordinal grafted, and a walk's depth), from a sealed document, so +//! every entry in them is genuine. What varies is which entries a dishonest +//! database hands back. + +mod common; + +use std::sync::atomic::Ordering; + +use common::CountingSource; +use serde_json::{json, Value}; +use stack_encrypt::json::{ + JsonDocument, JsonExtracted, JsonNode, JsonOptions, JsonPath, JsonSelector, JsonSubtree, + JsonSubtreeKey, +}; +use stack_encrypt::target::{AeadContext, CallerContext}; +use stack_encrypt::{nonempty, Error, StackCipher}; + +type Cipher = StackCipher; + +async fn cipher() -> (Cipher, std::sync::Arc) { + let source = CountingSource::new(); + let (_, retrieved) = source.counters(); + let cipher = StackCipher::builder() + .kms(source) + .init() + .await + .expect("build cipher"); + (cipher, retrieved) +} + +fn aead() -> AeadContext { + AeadContext::from(nonempty!("users").with("profile")) +} + +/// `$`, `hello` (1), `nested` (2), `deep` (3), `list` (4), its elements (5, +/// 7) and their `x` (6, 8), `number` (9): depth-first, members in key order. +fn fixture() -> Value { + json!({"hello": "world-1", "nested": {"deep": "constant", "list": [{"x": 1}, {"x": 2}]}, "number": 1}) +} + +async fn seal(cipher: &Cipher, value: &Value) -> JsonDocument { + cipher + .default_keyset() + .encrypt_as::( + value, + CallerContext::from(nonempty!("users").with("profile")), + ) + .await + .expect("seals") +} + +async fn selector(cipher: &Cipher, path: &str) -> JsonSelector { + cipher + .default_keyset() + .json_selector( + &JsonPath::parse(path).unwrap(), + nonempty!("users").with("profile"), + ) + .await + .unwrap() +} + +async fn key(cipher: &Cipher, path: &str) -> JsonSubtreeKey { + key_with(cipher, path, &JsonOptions::default()).await +} + +async fn key_with(cipher: &Cipher, path: &str, options: &JsonOptions) -> JsonSubtreeKey { + cipher + .default_keyset() + .json_subtree_key( + &JsonPath::parse(path).unwrap(), + options, + nonempty!("users").with("profile"), + ) + .await + .unwrap() +} + +async fn seal_with(cipher: &Cipher, value: &Value, options: &JsonOptions) -> JsonDocument { + cipher + .default_keyset() + .run( + stack_encrypt::target::json_document::<_, stack_encrypt::target::Borrowed>( + options.clone(), + ), + value, + CallerContext::from(nonempty!("users").with("profile")), + ) + .await + .expect("seals") +} + +/// The options with the root not sealed. +fn open_root() -> JsonOptions { + JsonOptions { + sealed_root: false, + ..JsonOptions::new() + } +} + +/// The entry at `j` as an extractor returns it, at depth `d` when a walk +/// returned it. +fn row(doc: &JsonDocument, j: usize, d: Option) -> JsonNode { + let mut wire = serde_json::to_value(doc.node(j).unwrap()).unwrap(); + if let Some(d) = d { + wire["d"] = json!(d); + } + serde_json::from_value(wire).unwrap() +} + +/// The rows the SQL walk returns for `key`, walked as it walks: with no +/// gates, every descendant of the token's node from depth 1; with gates, +/// through each gate (a selector, or `#n:`), then each node the +/// gates reach at depth 0 with everything beneath it. +fn walk(doc: &JsonDocument, key: &JsonSubtreeKey) -> Vec { + let gates = key.gates(); + if gates.is_empty() { + return doc + .subtree(key.token_bytes()) + .iter() + .map(|(j, _, d)| row(doc, *j, Some(*d))) + .collect(); + } + let mut frontier = vec![(None, *key.token_bytes())]; + for gate in &gates { + let mut next = Vec::new(); + for (_, token) in &frontier { + let children = doc.children(token); + match gate.strip_prefix('#') { + Some(slot) => { + let (n, wanted) = slot.split_once(':').unwrap(); + if let Some((j, entry, t)) = children.get(n.parse::().unwrap()) { + if base64(entry.selector()) == wanted { + next.push((Some(*j), *t)); + } + } + } + None => next.extend( + children + .iter() + .filter(|(_, entry, _)| base64(entry.selector()) == *gate) + .map(|(j, _, t)| (Some(*j), *t)), + ), + } + } + frontier = next; + } + let mut rows = Vec::new(); + for (j, token) in frontier { + rows.push(row(doc, j.unwrap(), Some(0))); + rows.extend( + doc.subtree(&token) + .iter() + .map(|(j, _, d)| row(doc, *j, Some(*d))), + ); + } + rows +} + +fn base64(bytes: &[u8]) -> String { + use base64ct::{Base64, Encoding}; + Base64::encode_string(bytes) +} + +async fn open_extracted(cipher: &Cipher, extracted: JsonExtracted) -> Result { + cipher + .decrypt_as::(extracted, aead()) + .await +} + +async fn open_value(cipher: &Cipher, rows: JsonSubtree) -> Result { + cipher.decrypt_as::(rows, aead()).await +} + +async fn open_list(cipher: &Cipher, rows: JsonSubtree) -> Result, Error> { + cipher + .decrypt_as::, JsonSubtree>(rows, aead()) + .await +} + +mod assertions { + use super::*; + + /// The read was refused as the wrong entry, naming `expected`. + pub(crate) fn wrong_entry(result: Result, expected: &str) { + match result { + Err(Error::JsonWrongEntry { expected: got }) => { + assert_eq!(got, expected, "the refusal names what the read expected") + } + other => panic!("expected JsonWrongEntry ({expected}), got {other:?}"), + } + } +} + +mod given_the_entry_arrow_returned { + use super::*; + + #[tokio::test] + async fn the_entry_at_the_path_opens_to_its_scalar() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let hello = selector(&cipher, "$.hello").await; + let opened = open_extracted(&cipher, hello.open(row(&doc, 1, None))).await; + assert_eq!(opened.unwrap(), json!("world-1"), "the value at the path"); + } + + #[tokio::test] + async fn another_paths_entry_is_refused_before_any_key_is_retrieved() { + let (cipher, retrieved) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let hello = selector(&cipher, "$.hello").await; + // `$.number` is a genuine entry of the row, and opens on its own + // selector: the database answered `$.hello` with it. + let extracted = hello.open(row(&doc, 9, None)); + assert_eq!(extracted.selector(), hello, "the selector asked for"); + assert_eq!(extracted.node().ordinal(), 9, "the row as returned"); + assert_wrong(open_extracted(&cipher, extracted).await); + assert_eq!( + retrieved.load(Ordering::SeqCst), + 0, + "the refusal retrieved no key" + ); + } + + #[tokio::test] + async fn the_sealed_root_is_refused_in_place_of_a_node() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let hello = selector(&cipher, "$.hello").await; + // The sealed root opens to the whole document: every field, for a + // read of one. + assert_wrong(open_extracted(&cipher, hello.open(row(&doc, 0, None))).await); + } + + #[tokio::test] + async fn the_sealed_root_opens_when_the_root_was_asked_for() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let root = selector(&cipher, "$").await; + let opened = open_extracted(&cipher, root.open(row(&doc, 0, None))).await; + assert_eq!(opened.unwrap(), fixture(), "`$` is the whole document"); + } + + #[tokio::test] + async fn an_entry_relabelled_with_the_selector_asked_for_fails_to_open() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let hello = selector(&cipher, "$.hello").await; + // Passing the check by rewriting `s` breaks the ciphertext's + // binding to its own selector instead. + let mut wire = serde_json::to_value(doc.node(9).unwrap()).unwrap(); + wire["s"] = json!(hello.to_string()); + let relabelled: JsonNode = serde_json::from_value(wire).unwrap(); + let opened = open_extracted(&cipher, hello.open(relabelled)).await; + assert!(matches!(opened, Err(Error::Aead)), "{opened:?}"); + } + + #[tokio::test] + async fn any_element_at_the_path_opens_since_a_selector_reads_a_position_as_the_wildcard() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let second = selector(&cipher, "$.nested.list[1].x").await; + // `->` returns the first element at the path; the selector cannot + // say which position was meant. + let opened = open_extracted(&cipher, second.open(row(&doc, 6, None))).await; + assert_eq!(opened.unwrap(), json!(1), "the first element's x"); + } + + #[tokio::test] + async fn the_same_path_from_another_row_opens_by_design() { + let (cipher, _) = cipher().await; + let other = seal(&cipher, &json!({"hello": "world-2"})).await; + let hello = selector(&cipher, "$.hello").await; + // Row provenance is the database's word (ADR 0010): an entry + // returned with its own row's header opens. + let opened = open_extracted(&cipher, hello.open(row(&other, 1, None))).await; + assert_eq!(opened.unwrap(), json!("world-2"), "another row's value"); + } + + fn assert_wrong(result: Result) { + assertions::wrong_entry(result, "the entry at the path asked for"); + } +} + +mod given_the_rows_of_a_walk_without_gates { + use super::*; + + const CHILD: &str = "a child of the walked node"; + const NEXT: &str = "the next node of the walked subtree"; + + #[tokio::test] + async fn the_honest_rows_open_to_the_nodes_value() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let nested = key(&cipher, "$.nested").await; + assert!(nested.gates().is_empty(), "a chain head's token"); + let rows = walk(&doc, &nested); + assert_eq!(rows.len(), 6, "nested's descendants"); + let value = open_value(&cipher, nested.open(rows.clone())).await; + assert_eq!(value.unwrap(), fixture()["nested"], "the sub-document"); + let list = open_list(&cipher, nested.open(rows)).await; + assert_eq!( + list.unwrap(), + vec![json!("constant"), json!([{"x": 1}, {"x": 2}])], + "the members' values" + ); + } + + #[tokio::test] + async fn the_rows_of_another_field_of_the_same_shape_are_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &json!({"a": {"x": 1}, "b": {"x": 2}})).await; + let a = key(&cipher, "$.a").await; + let b = key(&cipher, "$.b").await; + // The database answers `$.a` with `$.b`'s subtree: one genuine + // member, in its place, but under `$.b`'s path. + let refused = open_value(&cipher, a.open(walk(&doc, &b))).await; + assertions::wrong_entry(refused, CHILD); + } + + #[tokio::test] + async fn a_row_of_another_path_spliced_in_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let nested = key(&cipher, "$.nested").await; + // `number` (9) sits right after nested's subtree in the document, + // so it follows the rows without a gap; its selector is not a + // member of nested's. + let mut rows = walk(&doc, &nested); + rows.push(row(&doc, 9, Some(1))); + let refused = open_value(&cipher, nested.open(rows)).await; + assertions::wrong_entry(refused, CHILD); + } + + #[tokio::test] + async fn a_dropped_row_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let nested = key(&cipher, "$.nested").await; + // The first element (5) is dropped, so its `x` (6) would be taken + // for the element itself. + let rows: Vec = walk(&doc, &nested) + .into_iter() + .filter(|n| n.ordinal() != 5) + .collect(); + let refused = open_value(&cipher, nested.open(rows)).await; + assertions::wrong_entry(refused, NEXT); + } + + #[tokio::test] + async fn a_row_reported_at_the_wrong_depth_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let nested = key(&cipher, "$.nested").await; + let rows: Vec = walk(&doc, &nested) + .into_iter() + .map(|n| { + if n.ordinal() == 6 { + row(&doc, 6, Some(1)) + } else { + n + } + }) + .collect(); + let refused = open_value(&cipher, nested.open(rows)).await; + assertions::wrong_entry(refused, NEXT); + } + + #[tokio::test] + async fn a_member_reported_at_the_wrong_depth_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let nested = key(&cipher, "$.nested").await; + let rows: Vec = walk(&doc, &nested) + .into_iter() + .map(|n| { + if n.ordinal() == 3 { + row(&doc, 3, Some(2)) + } else { + n + } + }) + .collect(); + let refused = open_value(&cipher, nested.open(rows)).await; + assertions::wrong_entry(refused, CHILD); + } + + #[tokio::test] + async fn an_orphan_whose_parent_was_left_out_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let nested = key(&cipher, "$.nested").await; + // `deep` (3) and then the second element's `x` (8) alone: its + // parent and everything between are missing. + let rows = vec![row(&doc, 3, Some(1)), row(&doc, 8, Some(3))]; + let refused = open_value(&cipher, nested.open(rows)).await; + assertions::wrong_entry(refused, CHILD); + } + + #[tokio::test] + async fn a_gap_between_two_members_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &json!({"a": {"x": 1, "y": {"z": 2}, "w": 3}})).await; + let a = key(&cipher, "$.a").await; + // `$.a`'s members are w (2), x (3) and y (4) with z (5) below: the + // database leaves x out, and w and y are each genuine members. + let rows: Vec = walk(&doc, &a) + .into_iter() + .filter(|n| n.ordinal() != 3) + .collect(); + let refused = open_value(&cipher, a.open(rows)).await; + assertions::wrong_entry(refused, CHILD); + } + + #[tokio::test] + async fn the_deepest_document_the_index_seals_opens_from_the_open_root() { + let (cipher, _) = cipher().await; + // 127 arrays: the innermost scalar 127 levels below the root, the + // deepest the index seals. + let mut value = json!(1); + for _ in 0..127 { + value = json!([value]); + } + let doc = seal_with(&cipher, &value, &open_root()).await; + let root = key_with(&cipher, "$", &open_root()).await; + let rows = walk(&doc, &root); + assert_eq!(rows.len(), 127, "every node below the root"); + let opened = open_value(&cipher, root.open(rows)).await; + assert_eq!(opened.unwrap(), value, "the whole document"); + } + + #[tokio::test] + async fn a_subtree_cut_short_is_refused_as_the_wrong_entry() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let nested = key(&cipher, "$.nested").await; + // The last row is the second element's `x` (8): without it the + // element says it has a member the rows do not hold. + let mut rows = walk(&doc, &nested); + assert_eq!(rows.pop().map(|n| n.ordinal()), Some(8), "x, the last row"); + let refused = open_value(&cipher, nested.open(rows)).await; + assertions::wrong_entry(refused, NEXT); + } + + #[tokio::test] + async fn a_leading_element_left_out_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &json!({"arr": [10, 20, 30]})).await; + let arr = key(&cipher, "$.arr").await; + assert!(arr.gates().is_empty(), "a chain head's token"); + // The walk does not return the node it starts from, so nothing + // before the first row bounds where the rows begin. Leaving out the + // first element would open as `[20, 30]`, every element moved down + // one position. + let rows = walk(&doc, &arr); + let first = rows[0].ordinal(); + let rows: Vec = rows.into_iter().filter(|n| n.ordinal() != first).collect(); + assertions::wrong_entry(open_value(&cipher, arr.open(rows.clone())).await, CHILD); + assertions::wrong_entry(open_list(&cipher, arr.open(rows)).await, CHILD); + } + + #[tokio::test] + async fn trailing_children_left_out_are_not_detected_by_a_walk_without_gates() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &json!({"arr": [10, 20, 30]})).await; + let arr = key(&cipher, "$.arr").await; + // What the design leaves open until the walk returns the node it + // starts from (#1356): the node's child count is in the node's own + // entry, so the rows cannot say how many children follow the last + // one returned. The database withholding the last element opens as + // the shorter array. A member left out of an object is the same, + // wherever it sits, since nothing the client holds ranks the keys. + let mut rows = walk(&doc, &arr); + rows.pop(); + let opened = open_value(&cipher, arr.open(rows)).await; + assert_eq!(opened.unwrap(), json!([10, 20]), "the elements before it"); + } + + #[tokio::test] + async fn rows_with_no_reported_depth_are_placed_by_the_document() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let nested = key(&cipher, "$.nested").await; + // The depth is the database's report; the rows' positions and the + // opened nodes place them, so rows without it open the same. + let rows: Vec = walk(&doc, &nested) + .iter() + .map(|n| row(&doc, n.ordinal() as usize, None)) + .collect(); + let value = open_value(&cipher, nested.open(rows)).await; + assert_eq!(value.unwrap(), fixture()["nested"], "the sub-document"); + } +} + +mod given_the_rows_of_a_gated_walk { + use super::*; + + const REACHED: &str = "a node the walked path reaches"; + + #[tokio::test] + async fn the_honest_rows_open_to_the_reached_nodes() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let second = key(&cipher, "$.nested.list[1]").await; + let value = open_value(&cipher, second.open(walk(&doc, &second))).await; + assert_eq!(value.unwrap(), json!({"x": 2}), "the second element"); + let xs = key(&cipher, "$.nested.list[*].x").await; + let values = open_list(&cipher, xs.open(walk(&doc, &xs))).await; + assert_eq!(values.unwrap(), vec![json!(1), json!(2)], "every x"); + } + + #[tokio::test] + async fn another_position_of_the_array_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let second = key(&cipher, "$.nested.list[1]").await; + // The first element carries the same selector as the second: the + // position in its label tells them apart. + let first = key(&cipher, "$.nested.list[0]").await; + let refused = open_value(&cipher, second.open(walk(&doc, &first))).await; + assertions::wrong_entry(refused, REACHED); + } + + #[tokio::test] + async fn a_node_of_another_path_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let xs = key(&cipher, "$.nested.list[*].x").await; + let refused = open_list(&cipher, xs.open(vec![row(&doc, 3, Some(0))])).await; + assertions::wrong_entry(refused, REACHED); + } + + #[tokio::test] + async fn a_descendant_left_without_its_reached_node_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let list = key(&cipher, "$.nested.list").await; + // The list's own row is dropped: what is left are its children, + // which the gated walk does not return on their own. + let rows: Vec = walk(&doc, &list) + .into_iter() + .filter(|n| n.ordinal() != 4) + .collect(); + let refused = open_value(&cipher, list.open(rows)).await; + assertions::wrong_entry(refused, REACHED); + } + + #[tokio::test] + async fn a_reached_node_reported_below_depth_zero_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let xs = key(&cipher, "$.nested.list[*].x").await; + let refused = open_list(&cipher, xs.open(vec![row(&doc, 6, Some(1))])).await; + assertions::wrong_entry(refused, REACHED); + } + + #[tokio::test] + async fn a_subtree_cut_short_is_refused_as_the_wrong_entry() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let second = key(&cipher, "$.nested.list[1]").await; + // The element (7) says it has one member, and the database leaves + // that member (8) out: the same dropped row as a gap, at the end of + // the rows instead of between two of them. + let mut rows = walk(&doc, &second); + assert_eq!(rows.pop().map(|n| n.ordinal()), Some(8), "x, the last row"); + let refused = open_value(&cipher, second.open(rows)).await; + assertions::wrong_entry(refused, "the next node of the walked subtree"); + } + + #[tokio::test] + async fn the_sealed_root_among_walked_rows_is_refused() { + let (cipher, _) = cipher().await; + let doc = seal(&cipher, &fixture()).await; + let list = key(&cipher, "$.nested.list").await; + // The root at depth 1 is not the fallback, and its selector is not + // the list's. + let mut rows = walk(&doc, &list); + rows.push(row(&doc, 0, Some(1))); + let refused = open_value(&cipher, list.open(rows)).await; + assertions::wrong_entry(refused, REACHED); + } +}