Repository navigation
Conversation
… EncryptedJson
The JSON index could only be reached from typed Rust: a data plan parsed
`{"json": ...}` and refused it (json_index_pending), and the json_tree
family was missing from the EQL catalog, so no binding could name it.
The catalog now lists the family once, as a document family
(eql_domains::DOCUMENT_FAMILIES): its stored type is the target
`EncryptedJson` (public.eql_v4_json_tree), produced from the new `json`
value kind and carrying the `json` index, with EncryptedJsonQuery,
EncryptedJsonLeafQuery and EncryptedJsonEntry beside it. eql-codegen
renders the target table row, v4::all()/all_query() and the Go types from
it; the scalar SQL and struct materializers never see it, since its SQL
and Rust bodies are hand-written and v4 only.
A plan field names the target with the JSON index's options,
`"target": {"name": "EncryptedJson", "options": {...}}` (the bare name
for the defaults). The plan resolves the options once, when it is built,
so the field's writes, queries and reads run under one definition, and
refuses options on a target that takes none, a json field with no target,
and a sealed null on a document. TargetResolver::encrypt takes the
options; eql-bindings' encrypt_with seals the document through the same
JsonTree layout typed Rust uses.
The document crosses the binding as UTF-8 JSON text and is parsed once,
by json::parse_text: an object that names a member twice and an integer
literal outside the 64-bit range are refused rather than resolved or
rounded, and every refusal carries a code and a position, never the text
(TargetError::Json). It opens back to its text in a bytes leaf.
tests/json_tree_data_plan.rs holds the data plan to the typed layout: the
same selectors, ordering terms and containment projection for the same
document under the same column, each path opening what the other sealed,
and a batch of documents in one key request.
Refs #1347, #1348, #1349
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
encrypt/eql gains the generated EncryptedJSON, EncryptedJSONQuery,
EncryptedJSONLeafQuery and EncryptedJSONEntry. The Go-name rule, here and
in eql-codegen, spells the word Json as JSON wherever it is a word, not
only as a prefix (EncryptedJson is EncryptedJSON).
internal/record carries the `json` kind and a target's options
(Field.TargetOptions), written as the `{"name", "options"}` target the
engine reads. A hand-built plan can store and open a document through
the guest with the EQL types: json_tree_test.go round-trips scalar roots,
arrays, null and a u64, and shows a refused document is ErrEncoding with
the engine's code and none of the text. stashgen support follows.
Refs #1347, #1348, #1349
Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
🦋 Changeset detectedLatest commit: ba65680 The changes in this PR will be included in the next version bump. This PR includes changesets to release 12 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
This comment has been minimized.
This comment has been minimized.
Contributor
Mutation testing (cargo-mutants,
|
| caught | missed | unviable | timeout |
|---|---|---|---|
| 152 | 0 | 20 | 0 |
Every mutant in the changed lines was caught by a test.
This was referenced Oct 25, 2026
…son in the per-type PostgreSQL test The fuzz harness's TargetResolver still had encrypt's old signature, so fuzz regression (check-record) did not compile. The per-type PostgreSQL test failed on EncryptedJson, the first producible type with no plaintexts of its own: a document has operators of its own and is run in PostgreSQL with the generated fields. Refs #1347 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
…ions::json The mutants gate found the integer pre-scan's index arithmetic unpinned (a scan that left a string early passed while the number it reached fitted, and several replacements hung). The scan now walks slices with no arithmetic to mutate, and tests pin a long number inside a string, a number after an escaped quote, an unterminated string, a lone sign and a malformed token. The visitor drops visit_string, visit_none and visit_some, which serde_json never calls. wipe_value and TargetOptions::json get unit tests. Two survivors are excluded with their reason: the strict visitor's expecting and visit_unit (equivalent), and JsonPlaintext's Drop (no safe code can read a value after it is dropped; what it runs is wipe_value, now pinned). Refs #1347 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Contributor
Author
|
Automated code review of this pull request at
|
- json::parse_text is the engine's own strict parser, held to serde_json by a differential property test and a new fuzz:json-text target, so no heap block it frees holds the document (serde_json's scratch buffer and partial documents on refusal did); to_text sizes its buffer first. A heap-spy test watches the boundary. - check_source reads a json field's document by the engine's rules, so a malformed one is refused before any key, as encrypt refuses it. - A present "options": null is refused rather than read as defaults. - eql-bindings' query() refuses a document by name (DocumentQuery); a test pins the engine's and eql-bindings' rules for which targets take JSON options to agree. - Go: the per-type PostgreSQL test no longer counts EncryptedJson; opened JSON numbers compare as text; EQLGoName decodes a rune; stashgen names the missing document support instead of a kind mismatch. - Docs: ValueKind's break under Breaking, the Go CHANGELOG no longer points at internal/record, the generated eql doc sentence reads. Refs #1347 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Contributor
Author
|
Review response to the findings listed above (fixed in ffd4212 unless noted):
Found while fixing 2 and 3, and not fixed here: the JSON index's own seal path ( CI failures on the previous push (the |
This comment has been minimized.
This comment has been minimized.
…its wipes A local mutants run timed out on the parser's index arithmetic: every `at += 1` replaced by `at *= 1` loops forever, which the gate counts as a survivor. The parser now holds the unread text as a slice and moves by re-slicing, so a mutation changes what it reads rather than hanging. Tests pin a string's buffer to its raw length, an array's growth from four by doubling, and a container closed by the other bracket. JsonPlaintext's Drop is no longer excluded from mutation: heap_hygiene's new a_json_document_read_from_a_frame_leaves_no_plaintext_on_the_heap fails when it is a no-op. The strict-visitor exclusion goes with the visitor. Refs #1347 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
The local mutants run timed out on `len == capacity` replaced by `!=`: every push then doubled the buffer and wiped its ever larger spare capacity. Asking the buffer for its spare capacity leaves nothing there to mutate; the growth test still pins the sizes. Refs #1347 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
Contributor
Author
|
Follow-up to the response above. A local
|
…arbage CI's Linux run of EQL's heap_hygiene flagged a freed array buffer that held the marker in its unused slots: bytes another allocation freed before the watch began (the test's own format! growth, or the text-target test running beside it in the same process). The inputs are now built into strings sized before they are written, and EQL's boundary test, whose binary runs its tests as threads of one process, parses strings only; stack-encrypt's, run one test to a process, keeps containers. Refs #1347 Claude-Session: https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu
coderdan
added this pull request to stack #1377
October 25, 2026 22:47
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This lets a language SDK store an encrypted JSON document (EQL's
json_treetype) through a data plan. A data plan is the declaration a non-Rust SDK sends the Rust engine (Stack Encrypt) as data, field by field; until now the engine refused the JSON index there, and thejson_treefamily was missing from the EQL catalog, the one list every generated type comes from. EQL is the SQL library installed in the customer's PostgreSQL.A field now names the target
EncryptedJson(stored inpublic.eql_v4_json_tree), declares the newjsontype, gives the JSON index's options, and hands each document over as UTF-8 JSON text. The engine parses the text by one set of rules and seals it through exactly the layout typed Rust uses. This is slice A plus the full-document half of slice B of the plan in #1353; queries and reads follow in the next pull request.Changes
Catalog and generated code (
packages/eql)eql-domains: aDocumentFamilylist,DOCUMENT_FAMILIES = [JSON_TREE], naming the four domains and their roles (stored, query, leaf query, entry). Kept apart fromCATALOGso the scalar SQL and struct generators never see it: this family's SQL and Rust bodies are hand-written and EQL v4 only.eql-codegen: a target-table row forEncryptedJson(plaintextjson, indexjson, needleEncryptedJsonQuery);v4::all()/v4::all_query()list the document and its two query operands; the Go typesEncryptedJSON,EncryptedJSONQuery,EncryptedJSONLeafQuery,EncryptedJSONEntry. The Go-name rule spells the wordJsonasJSONwherever it is a word.eql-bindings:encryption::targets::encrypt_with(name, options, …)sealsEncryptedJsonunder the field's options;decryptopens it to its text. New errorsTargetError::Json,TargetError::OptionsandTargetError::DocumentQuery(a one-plaintextqueryof a document).Engine (
packages/stack-encrypt)jsonvalue kind (ValueKind::Json): a document crosses as text or bytes and opens back to its text as bytes (dynamic::read_json,dynamic::json_frame).json::parse_text, the one JSON parser at the boundary, and the engine's own (not serde_json's deserializer), so no heap block it frees holds the document: refuses an object that names a member twice, an integer literal outsidei64::MIN ..= u64::MAX, and text that is not JSON, each with a stable code and a position, never a byte of the text. Held to serde_json by a differential property test and a new fuzz target,fuzz:json-text.json::to_textsizes its buffer before it writes.check_sourcereads a document by these rules too, so a malformed one is refused before any key."target": {"name": "EncryptedJson", "options": {"sealed_root": …, "ordered": […]}}(or the bare name). Options are resolved once when the plan is built (TargetOptions::Json), so one definition serves writes, queries and reads. Refused: options on a target that takes none, ajsonfield with no target, a sealed null on a document.TargetResolver::encrypttakes the field's options (breaking for an outside resolver; CHANGELOG).Go (
languages/golang)internal/record: thejsonkind andField.TargetOptions.encrypt/json_tree_test.go: a document round trip (numbers compared as text) and refusals through the real WebAssembly guest (the Rust engine compiled for Go) with the EQL types.stashgennames the missing support for a field ofEncryptedJson(the next pull requests add it) instead of a kind mismatch; the per-type PostgreSQL test skips the document type rather than counting it.Docs: changeset for
@cipherstash/eql, CHANGELOG entries forstack-encrypt,eql-bindingsand the Go module, and a line indocs/plans/2026-10-17-json-tree-index.md.Assumptions
The five owner decisions in #1347 are not made; this stack takes the plan's recommended answer to each:
json.RawMessage(required) and*json.RawMessage(optional,null=sql). Here only the wire is fixed: a document crosses as text or bytes.json::parse_text); Go never builds a tree.Also assumed:
EncryptedJson(Goeql.EncryptedJSON), the plan's role-based name, over the existing SQL domainpublic.eql_v4_json_tree; issue json_tree is missing from the EQL catalog, so no Go types are generated for it #1348'sJSONTree…names were not used. The name lives in one catalog constant, so a rename is one line plus regeneration.{"json": …}beside"c"); a document is reached only through the target, as the plan says.[]byteit can wipe), and is accepted in a text or bytes leaf.table/column, and extended or row-bound contexts stay refused (plan section 5).Verification
macOS arm64, Rust 1.94.1, Go 1.26.10, on this branch. After the review fixes (ffd4212, d36fb16): the root
cargo clippyandcargo nextest run --workspace --all-features(1,474 passed before the last parser change;-p stack-encrypt922 passed after it),packages/eqlclippy and nextest,types:check, the four guest builds,wasm:guest:test,go:test,go:lint,go:test:postgresagainst PostgreSQL 17 in Docker with the EQL v4 bundle from the tree (passed),pnpm run test:scripts(81 files passed),fuzz:json-textfor 2 minutes (7.9M inputs, no failure), the new heap-spy tests (heap_hygienein stack-encrypt and EQL; each fails when its wipe is removed, checked), and a localcargo mutants --in-diffpre-screen of the parser. The original verification:cargo fmt --all --check(root,packages/eql, guest): clean.cargo clippy --locked --no-deps --workspace --all-targets --all-features -- -D warnings(root) andcargo clippy --workspace --all-targets --all-features --exclude eql_tests -- -D warnings(packages/eql): clean.mise x --env test -- cargo nextest run -p stack-encrypt --all-features: 905 passed (new:json::text,dynamic::json, thejsonkind, the target-options plan tests).packages/eql:cargo nextest run --workspace --exclude eql_tests: 620 passed, including the newtests/json_tree_data_plan.rs(batch round trip in one key request; data plan and typed layout write the same selectors, ordering terms and projection, and each opens the other's document; refusals before any key) andcargo test -p eql-codegen. The SQLx crate (eql_tests) was not built: it needs generated fixtures and a database.mise run types:generate, thenmise run types:check: clean after commit.wasm:guest:build,:deterministic,:eql,:eql:deterministicbuilt;mise run wasm:guest:testpassed (75 tests witheql).mise run go:test: passed, includingTestJSONDocumentsRoundTripThroughTheGuestandTestJSONTextThatIsNotADocumentIsErrEncoding.go generate ./...: no diff.Related
Stack
Merge order is bottom to top; each pull request is based on the one below it.
EncryptedJsonstored through a data plan (engine, EQL catalog, guest, Go runtime) (this pull request)https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu