feat(golang, stack-encrypt): optional sealed fields, stored as a sealed null or SQL NULL

.changeset/eql-bindings-optional-storage-only.md

@@ -0,0 +1,5 @@
+---
+"@cipherstash/eql": patch
+---
+
+`eql-bindings` (the Rust crate released with this package): an EQL v4 type whose terms a null has, a storage-only type (`Date`, `Integer`, `Text`, ...) or one whose only term is equality's (`DateEq`, `IntegerEq`, `TextEq`, ...), now also implements `EncryptFrom` and `DecryptInto` for an `Option` of its plaintext, and its query twin `EncryptFrom` for one. For a type of fixed width a `None` seals as a typed null as long as a value (vitaminc 0.8), so an optional column does not show which rows are absent. For text it seals as a null that is not padded, since text's ciphertext is as long as the text, so its length may show that the value is absent. Either way it opens as `None`; its `hm` is the one term every null shares, and the query of a `None` finds the stored nulls with `=`. A value seals exactly as before. The by-name dispatch (`encryption::targets`) reads, opens and queries these types through the `Option`; a type with an order or match term still refuses a null. eql-bindings' own `Date`, `Timestamp` and `Decimal` plaintexts declare their fixed width to vitaminc, so their `None` is padded too. The SQL bundle and the TypeScript types do not change.

languages/golang/CHANGELOG.md

@@ -10,6 +10,28 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
 
 ## [Unreleased]
 
+### Added
+
+- **An optional field: a sealed field declared over a pointer (#1298).**
+  `DOB *encrypt.Date` with `stash:"dob,encrypt_into=Date"` generates (it
+  was refused as a composite, #1293), and `Decrypt` gives `nil` back for a
+  `nil`. The new `null=` tag part says how `nil` is stored:
+  `null=sealed` seals it as a null as long as a value of the field's type,
+  so the column does not show which rows have no value; `null=sql` stores
+  nothing, so the field's columns are SQL `NULL` and its member in the
+  generated type is a pointer. `stashgen` picks `null=sealed` for a field
+  whose type has a fixed width (a bool, a number, `encrypt.Date`,
+  `time.Time`, `encrypt.Decimal`, the 128-bit integers) and whose only
+  index, if any, is equality, and asks for `null=sql` to be written on any
+  other: a null has an equality term and no other, and text shows its
+  length. Beside equality a sealed null stores the term every nil of the
+  field shares, and a generated `EqNull` method derives its `eq` query,
+  which finds the rows whose value is absent. A generated `EncryptNull`
+  method seals `nil` for an update of one such column, which writing SQL
+  `NULL` would not store. A stored null read into a
+  field that is not a pointer is an error from `Decrypt`. `-model` does not
+  map a `null=sql` field yet.
+
 ### Breaking
 
 - **Every order term changes: re-derive stored `ore`, `ope` and

languages/golang/cmd/stashgen/README.md

@@ -79,6 +79,7 @@ The first part of a tag is the field's name, which is the column name in a datab
 | `stash:"tenant,context_field"` | the field's value is the context of every other field in the struct, in place of a `context=` tag; stored as it is |
 | `stash:"-"` | leave the field out |
 | `` _ struct{} `stash:"context=documents,opaque"` `` | seal the struct as one value |
+| `stash:"born,encrypt_into=DateEq,null=sql"` | on a field declared over a pointer (`*encrypt.Date`): how it stores `nil`, `null=sealed` or `null=sql`; see "Optional fields" |
 
 A struct has one context: either a `_ struct{}` field with `context=`, or one field with `context_field`, not both.
 A `context_field` is a `string` whose value is a label such as `"tenants/acme"`, segments separated by `/`.
@@ -109,6 +110,26 @@ These words are the same as the Rust API's words for the same behaviour, with tw
 `opaque` seals the whole struct as one `bytes` field holding a JSON document (see "What crosses the binding").
 `json` names the JSON index, which the Rust API does not declare yet; this build refuses it.
 
+### Optional fields
+
+A sealed field declared over a pointer to one scalar (`*encrypt.Date`, `*int32`, `*string`, a pointer to a type defined over a scalar) is optional: `nil` is a value it may hold, and `Decrypt` gives it back as `nil`.
+`null=` says how the field stores `nil`. There are two ways, and they differ in what a reader of the database can see:
+
+| `null=` | Stored for `nil` | What a reader of the column sees |
+|---|---|---|
+| `sealed` | a sealed null, as long as a sealed value of the field's type, and beside an equality index the one term every null of the field shares | nothing in the ciphertext; in an equality column, that the absent rows are one group, as rows holding one repeated value are |
+| `sql` | nothing: the field's columns are `NULL`, and its member in the generated type is `nil` | which rows have no value |
+
+A sealed null hides that the value is absent only where its length matches a value's and no index beside it shows it, so it is the default for a field whose type has a fixed width (a `bool`, a number, `encrypt.Date`, `time.Time`, `encrypt.Decimal`, `encrypt.Int128` and `encrypt.Uint128`) and whose only index, if it has one, is `equality`: sealed with `encrypt`, `encrypt,index=equality`, or into an EQL type with no index or with equality alone (`Date`, `DateEq`, `Integer`, `IntegerEq`, ...).
+A null has an equality term, so its rows are one equality group and a query finds them: the generated `EqNull` derives the `eq` query of `nil`, which matches every row whose value is absent.
+A `null=sealed` field's entry in the generated `Fields` value also has `EncryptNull`, which seals `nil` for an update of that one column: a sealed null is a value, so set the column to what it returns, not to SQL `NULL`.
+A null has no order or match term, so a field with any other index, and a field of text or bytes, has to write `null=sql`: an order or match column would show which values are absent, and text shows its length, and `stashgen` will not choose a visible `NULL` for you.
+`null=sealed` on such a field stops `stashgen` with the reason.
+
+A `null=sql` field's member in the generated type is a pointer (`*eql.DateEq`, `*EncryptedUserScore`), `nil` when the value was; a `null=sealed` field's is the same type as a non-optional field's, and its column is never `NULL`.
+A stored null read into a field that is not a pointer is an error from `Decrypt`, naming the field, never a zero value.
+`-model` does not map a `null=sql` field yet, and `stashgen` stops if it is asked to.
+
 An embedded struct of your own adds its tagged fields to the outer struct.
 An embedded struct from another package takes one tag for all of its fields: `stash:",passthrough"` stores them as they are, and `stash:"-"` leaves them out.
 Tags for other libraries, such as `gorm`, `db` and `json`, are copied to the same field of the generated type.
@@ -254,10 +275,10 @@ Generated code sends the engine every sealed field with its value, under the dec
 A passthrough field stays on the host: the engine does nothing to it a program could observe, and the FFI codec cannot carry every Go type a program stores beside a ciphertext (a `driver.Valuer`, a `sql.NullTime`).
 An `opaque` struct crosses as one JSON document and is one column, decoded back into the exact Go types the struct declares; a field may be any type `encoding/json` carries both ways (a scalar or a type defined over one, `[]byte`, slices, maps with string or integer keys, pointers, structs whose fields are all exported, `time.Time`).
 A float in an opaque struct must be finite: `encoding/json` refuses NaN and the infinities, so `Encrypt` fails for the batch with `encrypt.ErrEncoding`. A sealed float field outside an opaque struct carries them.
-A sealed field outside an opaque struct is one scalar, or a type defined over one; a struct, slice or map field seals only inside an opaque struct.
+A sealed field outside an opaque struct is one scalar, a type defined over one, or a pointer to either (see "Optional fields"); a struct, slice or map field seals only inside an opaque struct.
 The scalars are `string`, `[]byte`, `bool`, every integer type (each sealed at its own width, `int` and `uint` as 64-bit), `encrypt.Int128` and `encrypt.Uint128`, `float32`, `float64`, `encrypt.Decimal`, `encrypt.Date` and `time.Time` (a timestamp). The wire type `stashgen` writes for each is the field's own: `int8` is `gensupport.Int8`, `time.Time` is `gensupport.Timestamp`, and so on.
 An order index (`ore`, `ope`) on a decimal, date or timestamp is refused: those order terms are not supported yet (#1236).
-The zero `encrypt.Date` is not a date: a date field must be set, and `Encrypt` refuses a zero one with `encrypt.ErrEncoding`, naming the field. A sealed field cannot be a pointer (`*encrypt.Date` is refused as a composite), so an optional date has no sealed form yet: keep it in an opaque struct, or store a sentinel date.
+The zero `encrypt.Date` is not a date: a date field must be set, and `Encrypt` refuses a zero one with `encrypt.ErrEncoding`, naming the field. An optional date is a `*encrypt.Date` field (see "Optional fields").
 A nil `[]byte` in a sealed field, or a nil slice of a type defined over `[]byte`, comes back from `Decrypt` as an empty, non-nil slice; inside an opaque struct, nil comes back as nil.
 A protobuf message with a `oneof` cannot be generated from a policy: protoc-gen-go puts its members in wrapper types, not in the message struct, and `Generate` refuses it.
 

languages/golang/cmd/stashgen/main_test.go

@@ -119,6 +119,48 @@ func TestRunFlags(t *testing.T) {
 	}
 }
 
+// A storage-only EQL type over a pointer (#1293): the generator takes
+// *encrypt.Date as it takes encrypt.Date, and stores nil as a sealed null,
+// as long as a date, since Date carries no index. From the reproduction in
+// #1293. Skips when the guest is not built, as the engine decides what an
+// EQL type is.
+func TestRunGeneratesNullableDOB(t *testing.T) {
+	checker, err := encrypt.NewChecker(context.Background())
+	if err != nil {
+		t.Skip(err)
+	}
+	_ = checker.Close()
+	for _, dateType := range []string{"encrypt.Date", "*encrypt.Date"} {
+		t.Run(dateType, func(t *testing.T) {
+			src := `package people
+
+import "github.com/cipherstash/stack/languages/golang/encrypt"
+
+type Person struct {
+	_ struct{} ` + "`stash:\"context=people\"`" + `
+	DOB ` + dateType + " `stash:\"dob,encrypt_into=Date\"`" + `
+}
+`
+			dir := writeModule(t, map[string]string{"model.go": src})
+			var stdout, stderr bytes.Buffer
+			if code := run([]string{"-type", "Person"}, dir, &stdout, &stderr, stashgen.GuestEngine); code != 0 {
+				t.Fatalf("exit %d, want 0\n%s", code, stderr.String())
+			}
+			written, err := os.ReadFile(filepath.Join(dir, "person_stash.go"))
+			if err != nil {
+				t.Fatal(err)
+			}
+			sealedNull := strings.Contains(string(written), `Null("dob", gensupport.NullSealed)`)
+			if pointer := strings.HasPrefix(dateType, "*"); sealedNull != pointer {
+				t.Fatalf("a %s field stores nil as a sealed null: %v, want %v\n%s", dateType, sealedNull, pointer, written)
+			}
+			if !strings.Contains(string(written), "DOB eql.Date") {
+				t.Fatalf("a sealed null keeps the column's type, eql.Date:\n%s", written)
+			}
+		})
+	}
+}
+
 // The real engine: the embedded guest, which the command links with the EQL
 // types, so a struct with encrypt_into=TextEq generates, and a type the
 // engine cannot produce yet is refused with nothing written. Skips when the

languages/golang/encrypt/ciphertext.go

@@ -15,7 +15,9 @@ import (
 // vitaminc-encrypt and must never scan or marshal where one belongs.
 //
 // A sealed field is one scalar — a string, a number, a bool or a []byte, or
-// a type defined over one — and seals as vitaminc's tagged leaf, which a
+// a type defined over one, or a pointer to either, whose nil is sealed as a
+// null as long as a value or stored as SQL NULL — and seals as vitaminc's
+// tagged leaf, which a
 // Rust record opens when its field is a Value (not a bare u32 or String). A struct, slice or map seals only as part of an opaque struct,
 // which crosses as one JSON document and is one leaf; stashgen refuses it
 // anywhere else, because the engine would seal it as a tree of leaves and a

languages/golang/encrypt/gensupport/codec.go

@@ -154,6 +154,10 @@ func (c *Codec[P, E]) Decrypt(ctx context.Context, d encrypt.Decrypter, encrypte
 			switch {
 			case f.verb == verbOmit:
 				continue
+			case !ok && f.null == NullSQL:
+				// The field was nil, so nothing was stored for it; it opens
+				// as nil.
+				continue
 			case !ok:
 				return nil, fmt.Errorf("gensupport: %s: value %d: Open gave no field %q", c.g.TypeName, i, f.name)
 			case f.verb == verbContextField:
@@ -190,10 +194,13 @@ func (c *Codec[P, E]) Decrypt(ctx context.Context, d encrypt.Decrypter, encrypte
 }
 
 // split sorts a value's fields into what crosses the binding and what stays:
-// every declared field must be present and nothing else may be.
+// every declared field must be present and nothing else may be. A nil field
+// declared with NullSQL is left out of the row, since nothing is stored for
+// it; one declared with NullSealed crosses as nil, which the engine seals.
 func (c *Codec[P, E]) split(vals Values) (record.Source, map[string]any, error) {
 	row := make(record.Source, len(c.plan.Fields))
 	keep := map[string]any{}
+	left := 0
 	for _, f := range c.g.Declaration.fields {
 		if f.verb == verbOmit {
 			continue
@@ -202,6 +209,10 @@ func (c *Codec[P, E]) split(vals Values) (record.Source, map[string]any, error)
 		if !ok {
 			return nil, nil, fmt.Errorf("the generated Source gave no field %q", f.name)
 		}
+		if v == nil && f.null == NullSQL {
+			left++
+			continue
+		}
 		if f.crosses() {
 			converted, err := asKind(f.kind, v)
 			if err != nil {
@@ -212,7 +223,7 @@ func (c *Codec[P, E]) split(vals Values) (record.Source, map[string]any, error)
 			keep[f.name] = v
 		}
 	}
-	if len(vals) != len(row)+len(keep) {
+	if len(vals) != len(row)+len(keep)+left {
 		for name := range vals {
 			if _, ok := row[name]; ok {
 				continue
@@ -297,6 +308,33 @@ func Get[T any](vals Values, name string) (T, error) {
 	return out, nil
 }
 
+// Deref is a field declared over a pointer as Source gives it: nil for a nil
+// pointer, else the value it points to.
+func Deref[T any](p *T) any {
+	if p == nil {
+		return nil
+	}
+	return *p
+}
+
+// GetPtr reads one opened field declared over a pointer: nil for a field
+// that opened as nil, which it stored as a null or as nothing, else a
+// pointer to the value as [Get] reads it.
+func GetPtr[T any](vals Values, name string) (*T, error) {
+	v, ok := vals[name]
+	if !ok {
+		return nil, fmt.Errorf("gensupport: the opened value has no field %q", name)
+	}
+	if v == nil {
+		return nil, nil
+	}
+	out, err := Get[T](vals, name)
+	if err != nil {
+		return nil, err
+	}
+	return &out, nil
+}
+
 // Records wraps a codec with a model's conversions, for separate columns:
 // EncryptRows and DecryptRows return and take the model.
 func Records[P, E, R any](codec *Codec[P, E], to func(E) R, from func(R) E) *RecordsCodec[P, R] {

languages/golang/encrypt/gensupport/declaration.go

@@ -76,8 +76,25 @@ type field struct {
 	verb     verb
 	indexes  []encrypt.Index
 	eqlType  string
+	null     NullMode
 }
 
+// NullMode is how a sealed field declared over a pointer (*T) stores nil:
+// stashgen's `null=` tag. A field with none is never nil.
+type NullMode string
+
+const (
+	// NullSealed seals nil like a value, as a null as long as a value of
+	// the field's kind, so the stored record does not show which rows are
+	// absent. Decrypt gives nil. An equality index stores the term every nil
+	// shares, which [Field.NullTerm] queries; the field has no other index,
+	// and its kind has a fixed width.
+	NullSealed NullMode = "sealed"
+	// NullSQL stores nothing for nil: the field's columns hold SQL NULL,
+	// which shows that the value is absent. Decrypt gives nil.
+	NullSQL NullMode = "sql"
+)
+
 // Declaration is a generated struct's declaration as data: its context and,
 // for each field, what happens to it. Only generated code builds one, from
 // the struct's stash tags, and only generated code reads it. A mistake in a
@@ -152,6 +169,35 @@ func (d Declaration) EncryptInto(name string, kind Kind, eqlType string) Declara
 	return d.add(field{name: name, kind: kind, verb: verbEncryptInto, eqlType: eqlType})
 }
 
+// Null says how the named field, declared over a pointer, stores nil. The
+// engine decides whether the field can, when the declaration is first used:
+// a sealed null needs a field whose only index is equality, of a kind of
+// fixed width.
+func (d Declaration) Null(name string, mode NullMode) Declaration {
+	if d.err != nil {
+		return d
+	}
+	if mode != NullSealed && mode != NullSQL {
+		d.err = fmt.Errorf("gensupport: Null(%q): unknown mode %q", name, mode)
+		return d
+	}
+	for i := range d.fields {
+		if d.fields[i].name != name {
+			continue
+		}
+		if !d.fields[i].sealed() {
+			d.err = fmt.Errorf("gensupport: Null(%q): only a sealed field stores nil", name)
+			return d
+		}
+		// A copy, so the receiver's fields are not changed under it.
+		d.fields = slices.Clone(d.fields)
+		d.fields[i].null = mode
+		return d
+	}
+	d.err = fmt.Errorf("gensupport: Null(%q): no such field", name)
+	return d
+}
+
 // Omit leaves the field out: it does not cross the binding and is not
 // stored. The declaration lists it so a reader sees the choice.
 func (d Declaration) Omit(name string) Declaration {
@@ -238,7 +284,7 @@ func (d Declaration) plan() (*record.Plan, error) {
 		if !f.sealed() {
 			continue
 		}
-		rf := record.Field{Name: f.name, Identity: f.identity, Kind: record.Kind(f.kind)}
+		rf := record.Field{Name: f.name, Identity: f.identity, Kind: record.Kind(f.kind), Null: record.Null(f.null)}
 		switch f.verb {
 		case verbEncryptInto:
 			// The EQL type's own plan decides the outputs; the engine

languages/golang/encrypt/gensupport/field.go

@@ -50,13 +50,33 @@ func (f Field[T]) Encrypt(ctx context.Context, c *encrypt.Cipher, v T) (Output,
 	if f.err != nil {
 		return Output{}, f.err
 	}
-	if c == nil {
-		return Output{}, fmt.Errorf("gensupport: %s: Encrypt needs a cipher", f.name)
-	}
 	value, err := asKind(f.kind(), v)
 	if err != nil {
 		return Output{}, fmt.Errorf("gensupport: %s: %w", f.name, err)
 	}
+	return f.seal(ctx, c, "Encrypt", value)
+}
+
+// EncryptNull seals nil for a field that stores nil as a sealed null: a
+// ciphertext as long as a value's, and the term every nil of the field
+// shares. It is the update that sets one such column to nil, which writing
+// SQL NULL is not. A field that stores nil any other way is refused: under
+// null=sql nil is SQL NULL, and a field of no null mode has no nil.
+func (f Field[T]) EncryptNull(ctx context.Context, c *encrypt.Cipher) (Output, error) {
+	if f.err != nil {
+		return Output{}, f.err
+	}
+	if pf := f.plan.Field(f.name); pf == nil || pf.Null != record.SealedNull {
+		return Output{}, fmt.Errorf("%w: gensupport: %s: EncryptNull seals nil only for a field that stores it as a sealed null (null=sealed)", encrypt.ErrEncoding, f.name)
+	}
+	return f.seal(ctx, c, "EncryptNull", nil)
+}
+
+// seal seals one value of the field, or nil, through the field's own plan.
+func (f Field[T]) seal(ctx context.Context, c *encrypt.Cipher, method string, value any) (Output, error) {
+	if c == nil {
+		return Output{}, fmt.Errorf("gensupport: %s: %s needs a cipher", f.name, method)
+	}
 	row := record.Source{f.name: value}
 	if f.plan.ContextField != "" {
 		// One value of one field has no record to take its context from:
@@ -100,6 +120,18 @@ func (f Field[T]) Range(ctx context.Context, c *encrypt.Cipher, operator string,
 	return terms[0], terms[1], nil
 }
 
+// NullTerm derives an exact operator's term for nil: for eq on a field that
+// stores nil as a sealed null, the term every such nil shares, so it finds
+// the rows whose value is absent. The engine refuses it for any other field
+// or operator: nothing else stores a null's term.
+func (f Field[T]) NullTerm(ctx context.Context, c *encrypt.Cipher, operator string) ([]byte, error) {
+	terms, err := f.query(ctx, c, operator, nil, 1)
+	if err != nil {
+		return nil, err
+	}
+	return terms[0], nil
+}
+
 // query derives an exact operator's terms, and refuses an answer of another
 // shape than the generated method's: a probabilistic term has a query type
 // of its own (Matches). Each operand is converted to the field's kind as a