feat(go): generate encrypted JSON document fields

languages/golang/CHANGELOG.md

@@ -12,6 +12,22 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
 
 ### Added
 
+- **Encrypted JSON documents with `stashgen`** (#1351, #1352). A
+  `json.RawMessage` field tagged `encrypt_into=JsonSearch` (options:
+  `ordered=$.a;$.b[*].c`, `sealed_root=false`) is stored as an
+  `eql.JSONSearch` in a `public.eql_v4_json_search` column; a
+  `*json.RawMessage` field takes `null=sql`. Its `Fields` entry has
+  `Contains`, `ContainedBy`, `EqAt`, `Selector`, `Subtree`, `Children` and
+  `ChildCount`, and `At.<Path>` per ordered path with `Lt`, `Lte`, `Gt`,
+  `Gte`, `Between` and the path's own `Selector`. `encrypt.JSONSelection`
+  and `encrypt.JSONRead` open the rows `->` and `eql_v4.json_tree_subtree`
+  return against the request that prepared them (`OpenOne` refuses a
+  wildcard read up front); `encrypt.JSONSelector`, `JSONToken` and
+  `JSONTokens` bind as `text` and `text[]`, and a zero selector or token is
+  refused rather than bound. A `json.RawMessage` sealed any other way stays
+  bytes. `gensupport.JSON`, `TargetOptions`,
+  `Declaration.Options` and the `JSON*` helpers are the generated code's.
+
 - **Query and read an encrypted JSON document, at the record level**
   (#1350, #1352). `Cipher.QueryTerm` derives a document field's operands:
   `QueryTerm.Encoding` says what the term holds, and `QueryTerm.Text`

languages/golang/cmd/stashgen/README.md

@@ -74,7 +74,8 @@ The first part of a tag is the field's name, which is the column name in a datab
 | `stash:"notes,encrypt"` | seal the field, with no index |
 | `stash:"email,encrypt,index=equality;match"` | seal the field, and derive each index beside it |
 | `stash:"score,index=ore"` | derive the index alone; no ciphertext is stored, so `Decrypt` leaves the field at its zero value |
-| `stash:"attrs,index=json"` | refused in this build: the engine does not derive the `json` index yet |
+| `stash:"profile,encrypt_into=JsonSearch(ordered=$.age)"` | on a `json.RawMessage`: store the field as an encrypted JSON document; see "Encrypted JSON documents" |
+| `stash:"attrs,index=json"` | refused: a JSON document is stored with `encrypt_into=JsonSearch`, not as an index beside a ciphertext |
 | `stash:"id,passthrough"` | store the field as it is |
 | `stash:"tenant,context_field"` | the field's value is the context of every other field in the struct, in place of a `context=` tag; stored as it is |
 | `stash:"-"` | leave the field out |
@@ -108,7 +109,36 @@ An index takes its options in parentheses after its name, separated by commas: `
 This build refuses an index with options: a query term is derived with the default options only, so a stored term with other options would never match one.
 These words are the same as the Rust API's words for the same behaviour, with two that only Go has.
 `opaque` seals the whole struct as one `bytes` field holding a storage blob (see "What crosses the binding").
-`json` names the JSON index, which the Rust API does not declare yet; this build refuses it.
+`json` names the JSON index, which a field reaches only through `encrypt_into=JsonSearch`.
+
+### Encrypted JSON documents
+
+A `json.RawMessage` field with `encrypt_into=JsonSearch` is stored as one encrypted JSON document, an `eql.JSONSearch` in a `public.eql_v4_json_search` column.
+The field holds the document's JSON text; the engine parses it, so an object that names a member twice, an integer outside the 64-bit range and text that is not JSON are refused with `encrypt.ErrEncoding`.
+Values come back as JSON text holding the same values; key order and the spelling of numbers are not kept.
+A `*json.RawMessage` field is optional and must say `null=sql`: a document has no fixed length to hide a missing one behind. A non-nil pointer to `null` is a stored JSON `null`.
+A `json.RawMessage` sealed any other way (`encrypt`, `index=equality`) is bytes, as any type defined over `[]byte` is.
+
+The type takes options in parentheses: `ordered=` names the paths whose numbers and strings carry an ordering term, separated by `;`, and `sealed_root=false` turns off the sealed root (on by default).
+`encrypt_into=JsonSearch(ordered=$.age;$.visits[*].at)` orders two paths.
+A path in a tag is plain JSONPath: `$`, then `.key` steps of letters, digits and underscores, and `[n]` or `[*]` steps; an ordered path names no position. The engine checks every path when `stashgen` runs.
+
+The field's `Fields` entry has a method per operation:
+
+| Method | Returns | SQL |
+|---|---|---|
+| `Contains(ctx, c, needle)`, `ContainedBy(ctx, c, needle)` | `eql.JSONSearchQuery` | `col @> $1::eql_v4.query_json`, `col <@ $1::eql_v4.query_json` |
+| `EqAt(ctx, c, path, v)` | `eql.JSONSearchQuery` | `col @> $1::eql_v4.query_json` |
+| `At.Age.Lt(ctx, c, v)`, `Lte`, `Gt`, `Gte`, `Between` | `eql.JSONSearchLeafQuery` | `(col -> $sel::text) < $1::eql_v4.query_json_leaf`, `$sel` from `At.Age.Selector(ctx, c)` |
+| `Selector(ctx, c, path)` | `encrypt.JSONSelection`: `Selector()` and `Open` | `col -> $1::text` |
+| `Subtree(ctx, c, path, opts)` | `encrypt.JSONRead`: `Tokens()`, `Open` and `OpenOne` | `eql_v4.json_tree_subtree(col, $1::text[], 64)` |
+| `Children(ctx, c, path)`, `ChildCount(ctx, c, path)` | `encrypt.JSONToken` | `eql_v4.json_tree_children(col, $1::text)`, `eql_v4.json_tree_child_count(col, $1::text)` |
+
+`At` has one accessor per ordered path, named for its keys (`$.visits[*].at` is `At.VisitsAt`, `$.a.1` is `At.A1`); two paths with one name, or one path given twice, stop `stashgen`.
+An ordering compares the value `->` extracts at its path, and EQL's `<` compares ordering terms alone, so take the selector from the same accessor (`At.Age.Selector`) rather than spelling the path again.
+`OpenOne` reads one value and refuses a read whose path has a wildcard before anything is opened; open a wildcard read with `Open`.
+The needle and the values are `json.RawMessage`: write the JSON you mean, such as `json.RawMessage(`+"`"+`{"active":true}`+"`"+`)`.
+A position in a path is refused for `EqAt` and `Selector`, whose selector reads every position as the wildcard; read a position with `Subtree`.
 
 ### Optional fields
 

languages/golang/cmd/stashgen/main_test.go

@@ -214,15 +214,15 @@ func TestRunAsksTheEmbeddedEngine(t *testing.T) {
 	if !strings.Contains(stderr.String(), "User.Age") || !strings.Contains(stderr.String(), "TextEq seals a string, and int64 is int") {
 		t.Fatalf("stderr = %q", stderr.String())
 	}
-	// A document type is refused by what is missing, not by a kind
-	// mismatch no Go type could fix.
+	// A document type over another Go type is refused by naming the Go
+	// types that carry one, not by a kind mismatch.
 	document := strings.Replace(userSource, "Email string   `stash:\"email,encrypt_into=TextEq\"`", "Profile []byte `stash:\"profile,encrypt_into=JsonSearch\"`", 1)
 	dir = writeModule(t, map[string]string{"model.go": document})
 	stderr.Reset()
 	if code := run([]string{"-type", "User"}, dir, &stdout, &stderr, stashgen.GuestEngine); code != 1 {
 		t.Fatalf("exit %d, want 1\n%s", code, stderr.String())
 	}
-	if !strings.Contains(stderr.String(), "User.Profile") || !strings.Contains(stderr.String(), "JsonSearch seals a json document, and stashgen does not generate a field of it yet") {
+	if !strings.Contains(stderr.String(), "User.Profile") || !strings.Contains(stderr.String(), "JsonSearch seals a json document: declare the field json.RawMessage") {
 		t.Fatalf("stderr = %q", stderr.String())
 	}
 	// A context of two segments leaves an EQL field no column: refused by
@@ -290,3 +290,84 @@ func TestModelFlagsRepeat(t *testing.T) {
 		t.Fatalf("String = %q", got)
 	}
 }
+
+// A json.RawMessage field stored as an encrypted JSON document, against the
+// embedded engine with the EQL types: the generated entry has the
+// document's methods and an accessor per ordered path, and every refusal
+// names the field before anything is written.
+func TestRunGeneratesJSONSearchFields(t *testing.T) {
+	checker, err := encrypt.NewChecker(context.Background())
+	if err != nil {
+		t.Skip(err)
+	}
+	_ = checker.Close()
+	source := func(fields string) string {
+		return "package users\n\nimport \"encoding/json\"\n\nvar _ json.RawMessage\n\ntype Customer struct {\n\t_ struct{} `stash:\"context=customers\"`\n\tID int64 `stash:\"id,passthrough\"`\n" + fields + "\n}\n"
+	}
+	generate := func(fields string) (string, string, int) {
+		dir := writeModule(t, map[string]string{"model.go": source(fields)})
+		var stdout, stderr bytes.Buffer
+		code := run([]string{"-type", "Customer"}, dir, &stdout, &stderr, stashgen.GuestEngine)
+		written, _ := os.ReadFile(filepath.Join(dir, "customer_stash.go"))
+		return string(written), stderr.String(), code
+	}
+	written, stderr, code := generate("\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$.age;$.visits[*].at,sealed_root=false)\"`\n\tPrefs *json.RawMessage `stash:\"prefs,encrypt_into=JsonSearch,null=sql\"`")
+	if code != 0 {
+		t.Fatalf("exit %d\n%s", code, stderr)
+	}
+	for _, want := range []string{
+		"Profile eql.JSONSearch",
+		"Prefs   *eql.JSONSearch",
+		`EncryptInto("profile", gensupport.JSON, "JsonSearch").`,
+		`Options("profile", gensupport.TargetOptions{SealedRoot: gensupport.Ptr(false), Ordered: []string{"$.age", "$.visits[*].at"}})`,
+		"func (f ProfileField) Contains(ctx context.Context, c *encrypt.Cipher, needle json.RawMessage) (eql.JSONSearchQuery, error)",
+		"func (f ProfileField) Subtree(ctx context.Context, c *encrypt.Cipher, path string, opts encrypt.JSONReadOptions) (encrypt.JSONRead[eql.JSONEntry], error)",
+		"Age      ProfileFieldPath // $.age",
+		"VisitsAt ProfileFieldPath // $.visits[*].at",
+		"func (p ProfileFieldPath) Between(ctx context.Context, c *encrypt.Cipher, lo, hi json.RawMessage) (encrypt.JSONRange[eql.JSONSearchLeafQuery], error)",
+	} {
+		if !strings.Contains(written, want) {
+			t.Errorf("the generated file lacks %q", want)
+		}
+	}
+	if strings.Contains(written, "PrefsFieldAt") {
+		t.Error("Prefs orders no path, and got an At")
+	}
+	if !strings.Contains(written, "func (p ProfileFieldPath) Selector(ctx context.Context, c *encrypt.Cipher) (encrypt.JSONSelection[eql.JSONEntry], error)") {
+		t.Error("an ordered path has no Selector of its own")
+	}
+	// A json.RawMessage sealed any other way is bytes, as it was before
+	// documents; an object key of digits names its accessor.
+	written, stderr, code = generate("\tData json.RawMessage `stash:\"data,encrypt,index=equality\"`\n\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$.a.1)\"`")
+	if code != 0 {
+		t.Fatalf("exit %d\n%s", code, stderr)
+	}
+	for _, want := range []string{`EncryptIndex("data", gensupport.Bytes`, "A1 ProfileFieldPath // $.a.1"} {
+		if !strings.Contains(written, want) {
+			t.Errorf("the generated file lacks %q", want)
+		}
+	}
+	for _, c := range []struct{ name, fields, want string }{
+		{"a sealed null", "\tPrefs *json.RawMessage `stash:\"prefs,encrypt_into=JsonSearch,null=sealed\"`", "Customer.Prefs"},
+		{"a nil document that does not say null=sql", "\tPrefs *json.RawMessage `stash:\"prefs,encrypt_into=JsonSearch\"`", "null=sql"},
+		{"a position in an ordered path", "\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$.scores[2])\"`", "the engine refuses the declaration"},
+		{"a quoted key in a tag", "\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$[\\\"a b\\\"])\"`", "not a plain JSONPath"},
+		{"an unknown option", "\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(bound=3)\"`", "no option \"bound\""},
+		{"the Go name in the tag", "\tProfile json.RawMessage `stash:\"profile,encrypt_into=JSONSearch\"`", "as the engine does, JsonSearch"},
+		{"two paths, one accessor", "\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$.a_b;$.aB)\"`", "both name the accessor At.AB"},
+		{"the same ordered path twice", "\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$.a;$.a)\"`", "the ordered path $.a is given twice"},
+		{"a path with no accessor name", "\tProfile json.RawMessage `stash:\"profile,encrypt_into=JsonSearch(ordered=$.__)\"`", "no Go accessor name"},
+		{"a string into a document", "\tProfile string `stash:\"profile,encrypt_into=JsonSearch\"`", "JsonSearch seals a json"},
+		{"options on a type that takes none", "\tEmail string `stash:\"email,encrypt_into=TextEq(ordered=$.a)\"`", "the engine refuses the declaration"},
+	} {
+		t.Run(c.name, func(t *testing.T) {
+			written, stderr, code := generate(c.fields)
+			if code != 1 || written != "" {
+				t.Fatalf("exit %d, wrote %d bytes\n%s", code, len(written), stderr)
+			}
+			if !strings.Contains(stderr, c.want) {
+				t.Fatalf("stderr = %q, want %q", stderr, c.want)
+			}
+		})
+	}
+}

languages/golang/encrypt/README.md

@@ -576,7 +576,7 @@ the installer disables these domains, and the `Ord` types are the ones to
 use. Text whose fold is longer than 1023 bytes, or ends in NUL, has no block
 ORE term and is refused, stored or queried.
 
-The rest of the catalog (ordered dates, timestamps and decimals, JSON) is listed in
+The rest of the catalog (ordered dates, timestamps and decimals, the v3 JSON types) is listed in
 `eql.Types` with the reason it is not produced yet, and `stashgen` refuses it
 with that reason. The guest builds the value: the generated code
 imports `encrypt/eql`, which embeds the build of the engine that holds the
@@ -591,6 +591,34 @@ with a tenant part refuses a struct with an `encrypt_into` field: an EQL
 value is stored under a table and a column, and the extended label has no
 column.
 
+### Encrypted JSON documents
+
+`JsonSearch` stores a whole JSON document in one `public.eql_v4_json_search`
+column, from a `json.RawMessage` field (`stash:"profile,encrypt_into=JsonSearch(ordered=$.age)"`;
+see the stashgen README). One entry per node is encrypted, so the database
+can test containment, equality and ordering at a path, return one node with
+`->`, and walk a subtree, without decrypting:
+
+```go
+q, err := users.Fields.Profile.Contains(ctx, cipher, json.RawMessage(`{"plan":"pro"}`))
+rows, err := db.QueryContext(ctx, `SELECT id FROM users WHERE profile @> $1::eql_v4.query_json`, q)
+
+read, err := users.Fields.Profile.Subtree(ctx, cipher, "$.visits[*].at", encrypt.JSONReadOptions{})
+// SELECT e FROM users, eql_v4.json_tree_subtree(profile, $1::text[], 64) AS e WHERE id = $2
+// with read.Tokens() as $1; collect each row's entries, then:
+values, err := read.Open(ctx, entriesPerRow)
+```
+
+A selection or read opens rows only as answers to the request that prepared
+it: the engine derives the selector or the read again from the field, the
+path and the cipher's keyset, and refuses a row that is not one it asked
+for. It does not prove the database returned every row, or rows of the
+database row you meant. `encrypt.JSONTokens` binds as a `text[]` literal,
+and a selector or token as `text`; cast each parameter, as above, so
+PostgreSQL picks EQL's operator and not `jsonb`'s. Numbers order as `float64`,
+so integers past 2^53 can tie at an ordered path; containment and `EqAt`
+compare them exactly.
+
 ## Building
 
 The package embeds `wasm/stack_encrypt_guest.wasm`, a build artefact of the

languages/golang/encrypt/eql_postgres_test.go

@@ -114,8 +114,9 @@ func TestEQLTypesRoundTripAndQueryInPostgres(t *testing.T) {
 			continue
 		}
 		// A document has operators of its own, not one query per index,
-		// so it is not run here or counted: it is run in PostgreSQL with
-		// the generated fields that store it.
+		// so it is not run here or counted:
+		// TestJSONDocumentsQueryAndReadInPostgres runs it with the
+		// generated fields that store it.
 		if typ.Plaintext == "json" {
 			continue
 		}

languages/golang/encrypt/gensupport/codec.go

@@ -26,7 +26,7 @@ type Output struct {
 	Match      encrypt.MatchTerm
 	Ore        encrypt.OreTerm
 	Ope        encrypt.OpeTerm
-	JSON       encrypt.JSONTerm
+	JSON       encrypt.JSONTerm //nolint:staticcheck // the deprecated output, kept so a declaration naming it fails into a Go type
 	EQL        []byte
 }
 

languages/golang/encrypt/gensupport/declaration.go

@@ -48,6 +48,9 @@ const (
 	Timestamp Kind = "timestamp"
 	String    Kind = "string"
 	Bytes     Kind = "bytes"
+	// JSON is a json.RawMessage field stored as a JSONSearch document:
+	// it crosses as its JSON text, which the engine parses.
+	JSON Kind = "json"
 )
 
 // OpaqueField is the one field of an opaque declaration: the whole struct,
@@ -78,6 +81,19 @@ type field struct {
 	indexes  []encrypt.Index
 	eqlType  string
 	null     NullMode
+	options  *TargetOptions
+}
+
+// TargetOptions are the options an EncryptInto field gives its EQL type:
+// for JsonSearch, the JSON index's. The engine checks them when the
+// declaration is first used (stashgen asked it when the file was written).
+type TargetOptions struct {
+	// SealedRoot says whether the document's root is sealed; nil is the
+	// default, sealed.
+	SealedRoot *bool
+	// Ordered are the JSONPath paths whose number and string leaves carry
+	// an ordering term.
+	Ordered []string
 }
 
 // NullMode is how a sealed field declared over a pointer (*T) stores nil:
@@ -170,6 +186,28 @@ func (d Declaration) EncryptInto(name string, kind Kind, eqlType string) Declara
 	return d.add(field{name: name, kind: kind, verb: verbEncryptInto, eqlType: eqlType})
 }
 
+// Options gives the named EncryptInto field's EQL type its options: the
+// `encrypt_into=JsonSearch(ordered=...)` tag.
+func (d Declaration) Options(name string, options TargetOptions) Declaration {
+	if d.err != nil {
+		return d
+	}
+	for i := range d.fields {
+		if d.fields[i].name != name {
+			continue
+		}
+		if d.fields[i].verb != verbEncryptInto {
+			d.err = fmt.Errorf("gensupport: Options(%q): only an EncryptInto field's EQL type takes options", name)
+			return d
+		}
+		d.fields = slices.Clone(d.fields)
+		d.fields[i].options = &options
+		return d
+	}
+	d.err = fmt.Errorf("gensupport: Options(%q): no such field", name)
+	return d
+}
+
 // Null says how the named field, declared over a pointer, stores nil. The
 // engine decides whether the field can, when the declaration is first used:
 // a sealed null needs a field whose only index is equality, of a kind of
@@ -293,6 +331,9 @@ func (d Declaration) plan() (*record.Plan, error) {
 			// is the engine's to say, at the first call (stashgen asked it
 			// when the file was written).
 			rf.Target = f.eqlType
+			if f.options != nil {
+				rf.TargetOptions = &record.TargetOptions{SealedRoot: f.options.SealedRoot, Ordered: slices.Clone(f.options.Ordered)}
+			}
 		case verbEncrypt:
 			rf.Outputs = []record.Output{record.Ciphertext}
 		case verbEncryptIndex:
@@ -320,3 +361,7 @@ func (d Declaration) plan() (*record.Plan, error) {
 var termOutputs = map[record.Output]struct{}{
 	record.Equality: {}, record.Match: {}, record.Ore: {}, record.Ope: {},
 }
+
+// Ptr is a pointer to v: what a generated TargetOptions literal spells an
+// option set to false with.
+func Ptr[T any](v T) *T { return &v }

languages/golang/encrypt/gensupport/gensupport_internal_test.go

@@ -73,7 +73,7 @@ func TestDeclarationRefusals(t *testing.T) {
 		"identity of no field":      Declare("u").Encrypt("a", String).Identity("b", "x"),
 		"seals nothing":             Declare("u").Passthrough("id"),
 		"EQL type unnamed":          Declare("u").EncryptInto("email", String, ""),
-		"json index":                Declare("u").EncryptIndex("a", String, encrypt.JSON()),
+		"json index":                Declare("u").EncryptIndex("a", String, encrypt.JSON()), //nolint:staticcheck // the deprecated index is still refused
 		"name not a label":          Declare("u").Encrypt("1a", String),
 	}
 	for name, d := range cases {