feat(go): seal an opaque struct as a vitaminc storage blob

languages/golang/CHANGELOG.md

@@ -42,6 +42,43 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
 
 ### Breaking
 
+- **An opaque struct seals as a storage blob, and reads the JSON it used
+  to (#1347).** `stash:"context=...,opaque"` now seals the struct as one
+  blob in vitaminc's storage profile, version 1 (`vcffi.MarshalStorage`),
+  the format vitaminc keeps readable across releases and languages, where
+  it sealed the JSON document `encoding/json` wrote. Rows sealed before
+  still decrypt: `Decrypt` reads a value that is not a blob as JSON.
+  **Run stashgen again:** generated files name
+  `gensupport.GeneratedVersion2`, so a file generated before does not
+  compile against this release (it would seal blobs keyed by Go field
+  names that the regenerated file could not read). The generated shape
+  struct names each field `vc:"name,default"`, so a field added to the
+  struct later reads as its zero value from a blob sealed before it, as
+  it did from JSON. A field added to a nested struct type has no such
+  tag: give it a pointer type, or rows sealed before it do not decrypt.
+  **Upgrade every reader before any writer:** an earlier release cannot
+  read a blob, so a rolling deploy or a rollback that runs an earlier
+  release after this one has sealed a row fails to decrypt that row.
+  What else changes for a program: a NaN or infinite float now seals
+  (JSON refused it with `ErrEncoding`); a nil slice or map comes back
+  empty, not nil, from a blob; `Encrypt` refuses, as `ErrEncoding`
+  wrapping `vcffi.ErrUnwritable`, a string that is not UTF-8 (JSON wrote
+  U+FFFD), a value nested more than 128 levels deep, and two map keys
+  whose `MarshalText` spells alike. `stashgen` refuses in an opaque
+  struct `encrypt.Decimal` and `encrypt.Date` (the storage profile has no
+  form for them), a type with only one of `MarshalText` and
+  `UnmarshalText`, a map key that is not an integer, a string, bytes or
+  text, and a type that only `MarshalJSON` carries over a struct with
+  unexported fields. **Rows already sealed with an `encrypt.Decimal` or
+  `encrypt.Date` in an opaque struct do not decrypt under this release**,
+  since no generated file can name the field: before upgrading, decrypt
+  them with the earlier release, change the field to a string (or to its
+  parts), and encrypt them again; the earlier release writes JSON, which
+  this release reads. A blob a later release writes is
+  `encrypt.ErrStorageVersion`. This takes vcffi from the unreleased
+  cipherstash/vitaminc#526 by pseudo-version; the storage format is a
+  compatibility commitment only once vitaminc releases it, so no Go
+  release should be cut while the pin is a pseudo-version.
 - **Every order term changes: re-derive stored `ore`, `ope` and
   `ore_block` index terms.** The engine moved to vitaminc 0.8, clww-ore 0.6
   (formerly cllw-ore) and ore-rs 0.10, and both order schemes now derive

languages/golang/cmd/stashgen/README.md

@@ -107,7 +107,7 @@ An EQL `TextMatch` field is stricter: see below.
 An index takes its options in parentheses after its name, separated by commas: `index=equality;match(k=3)`.
 This build refuses an index with options: a query term is derived with the default options only, so a stored term with other options would never match one.
 These words are the same as the Rust API's words for the same behaviour, with two that only Go has.
-`opaque` seals the whole struct as one `bytes` field holding a JSON document (see "What crosses the binding").
+`opaque` seals the whole struct as one `bytes` field holding a storage blob (see "What crosses the binding").
 `json` names the JSON index, which the Rust API does not declare yet; this build refuses it.
 
 ### Optional fields
@@ -227,7 +227,7 @@ The error names the type and the field, and never a value.
 - an embedded struct from another package with no tag;
 - a struct whose every field is left out;
 - a struct, slice or map field with `encrypt` or `index=` outside an opaque struct;
-- a field of an opaque struct whose type JSON cannot carry both ways.
+- a field of an opaque struct whose type the storage profile or JSON cannot carry both ways.
 
 ## Declarations from a policy
 
@@ -273,13 +273,21 @@ No warning, error or log line holds a plaintext value.
 
 Generated code sends the engine every sealed field with its value, under the declaration lowered to data: each field's label (`<context>/<name>`), its outputs and its wire type (`int64`, `string`, `bytes`, ...), which `stashgen` chose from the field's Go type.
 A passthrough field stays on the host: the engine does nothing to it a program could observe, and the FFI codec cannot carry every Go type a program stores beside a ciphertext (a `driver.Valuer`, a `sql.NullTime`).
-An `opaque` struct crosses as one JSON document and is one column, decoded back into the exact Go types the struct declares; a field may be any type `encoding/json` carries both ways (a scalar or a type defined over one, `[]byte`, slices, maps with string or integer keys, pointers, structs whose fields are all exported, `time.Time`).
-A float in an opaque struct must be finite: `encoding/json` refuses NaN and the infinities, so `Encrypt` fails for the batch with `encrypt.ErrEncoding`. A sealed float field outside an opaque struct carries them.
+An `opaque` struct crosses as one storage blob and is one column, read back into the exact Go types the struct declares.
+The blob is vitaminc's storage profile, version 1 (`vcffi.MarshalStorage`), the format vitaminc keeps readable across releases and languages.
+A field may be any type that both the storage profile and `encoding/json` carry both ways: a scalar or a type defined over one, `[]byte`, slices, maps with string or integer keys, pointers, structs whose fields are all exported, and types with both `MarshalText` and `UnmarshalText`, such as `time.Time`.
+`encrypt.Decimal` and `encrypt.Date` have no storage form and are refused in an opaque struct: store them as text, or by their parts.
+A row an earlier release sealed with one of them in an opaque struct does not decrypt under this release, because no generated file can name the field: decrypt such rows with the earlier release, change the field's type, and encrypt them again before upgrading.
+The generated shape struct tags each field `vc:"name,default"`, so a field added to the opaque struct reads as its zero value from a row sealed before it; a field added to a nested struct type needs a pointer type for the same.
+`Encrypt` refuses a string that is not UTF-8 and a value nested more than 128 levels deep, as `encrypt.ErrEncoding`.
+`encoding/json` still matters because a row an earlier release sealed holds a JSON document, and `Decrypt` reads it as JSON.
+A float in an opaque struct may be NaN or an infinity: the blob writes a float by its bits.
+A nil slice or map in an opaque struct comes back empty, not nil, from a blob; from an earlier release's JSON document it comes back nil.
 A sealed field outside an opaque struct is one scalar, a type defined over one, or a pointer to either (see "Optional fields"); a struct, slice or map field seals only inside an opaque struct.
 The scalars are `string`, `[]byte`, `bool`, every integer type (each sealed at its own width, `int` and `uint` as 64-bit), `encrypt.Int128` and `encrypt.Uint128`, `float32`, `float64`, `encrypt.Decimal`, `encrypt.Date` and `time.Time` (a timestamp). The wire type `stashgen` writes for each is the field's own: `int8` is `gensupport.Int8`, `time.Time` is `gensupport.Timestamp`, and so on.
 A decimal orders by value, so `1.5` ties `1.50`; a `time.Time` orders by the microsecond, so instants within one microsecond tie.
 The zero `encrypt.Date` is not a date: a date field must be set, and `Encrypt` refuses a zero one with `encrypt.ErrEncoding`, naming the field. An optional date is a `*encrypt.Date` field (see "Optional fields").
-A nil `[]byte` in a sealed field, or a nil slice of a type defined over `[]byte`, comes back from `Decrypt` as an empty, non-nil slice; inside an opaque struct, nil comes back as nil.
+A nil `[]byte` in a sealed field, or a nil slice of a type defined over `[]byte`, comes back from `Decrypt` as an empty, non-nil slice; inside an opaque struct it comes back empty from a blob too, and nil only from an earlier release's JSON document.
 A protobuf message with a `oneof` cannot be generated from a policy: protoc-gen-go puts its members in wrapper types, not in the message struct, and `Generate` refuses it.
 
 ## Status

languages/golang/encrypt/ciphertext.go

@@ -19,7 +19,7 @@ import (
 // null as long as a value or stored as SQL NULL — and seals as vitaminc's
 // tagged leaf, which a
 // Rust record opens when its field is a Value (not a bare u32 or String). A struct, slice or map seals only as part of an opaque struct,
-// which crosses as one JSON document and is one leaf; stashgen refuses it
+// which crosses as one storage blob and is one leaf; stashgen refuses it
 // anywhere else, because the engine would seal it as a tree of leaves and a
 // column holds one.
 type Ciphertext []byte

languages/golang/encrypt/doc.go

@@ -131,8 +131,9 @@
 // Passthrough fields never cross the binding: the engine does nothing to a
 // passthrough value that a program could observe, and the FFI codec cannot
 // carry every Go type a program stores beside a ciphertext (a
-// driver.Valuer, a sql.NullTime). An opaque struct crosses as one JSON
-// document and is one column.
+// driver.Valuer, a sql.NullTime). An opaque struct crosses as one storage
+// blob (vitaminc's storage profile, vcffi.MarshalStorage) and is one column;
+// a row an earlier release sealed as a JSON document still opens.
 //
 // # Terms
 //

languages/golang/encrypt/errors.go

@@ -1,6 +1,9 @@
 package encrypt
 
-import "github.com/cipherstash/stack/languages/golang/internal/guest"
+import (
+	"github.com/cipherstash/stack/languages/golang/internal/guest"
+	"github.com/cipherstash/vitaminc/bindings/go/vcffi"
+)
 
 // Failure kinds surfaced across the boundary: the guest reports a status
 // code, and these are what the codes decode to. They separate a tampered
@@ -66,6 +69,12 @@ var (
 	ErrMemoryLock = guest.ErrMemoryLock
 )
 
+// ErrStorageVersion is an opaque struct sealed as a storage blob of a
+// version this release does not read: a later release wrote it. Its error
+// wraps [ErrEncoding] too. It is vcffi.ErrUnsupportedVersion, named here so
+// a program can check for it without importing vitaminc's bindings.
+var ErrStorageVersion = vcffi.ErrUnsupportedVersion
+
 // Diagnostic is the full error behind a failure the guest reports, beside
 // the kind: every such failure is a *Diagnostic wrapping one of the
 // sentinels above, so errors.Is matches the kind and errors.As reads the

languages/golang/encrypt/gensupport/codec.go

@@ -1,13 +1,15 @@
 package gensupport
 
 import (
+	"bytes"
 	"context"
 	"encoding/json"
 	"fmt"
 	"sync"
 
 	"github.com/cipherstash/stack/languages/golang/encrypt"
 	"github.com/cipherstash/stack/languages/golang/internal/record"
+	"github.com/cipherstash/vitaminc/bindings/go/vcffi"
 )
 
 // Values is a struct's field values by declared name: what Source gives and
@@ -377,23 +379,32 @@ func (c *RecordsCodec[P, R]) Decrypt(ctx context.Context, d encrypt.Decrypter, r
 }
 
 // opaqueBytes is an opaque struct's fields as the one value the engine
-// seals: a JSON document of the generated shape struct, so the struct is one
-// column and every field type encoding/json round-trips comes back as it
-// was.
+// seals: a storage blob (vitaminc's storage profile, version 1, written by
+// vcffi.MarshalStorage) of the generated shape struct, so the struct is one
+// column and every field comes back at the Go type the struct declares. The
+// storage profile is the format for bytes that are kept: unlike the
+// transport, its version 1 is a compatibility commitment shared with Rust,
+// and it carries every float, NaN and the infinities included.
 func opaqueBytes(fields any) ([]byte, error) {
-	encoded, err := json.Marshal(fields)
+	encoded, err := vcffi.MarshalStorage(fields)
 	if err != nil {
-		// encoding/json refuses NaN and the infinities, which a sealed float
-		// field outside an opaque struct accepts.
-		// Its error is not wrapped: encoding/json's text quotes the value.
-		return nil, fmt.Errorf("%w: the opaque value does not encode as JSON (NaN, an infinity, or a type encoding/json refuses)", encrypt.ErrEncoding)
+		// vcffi's error names a type or a kind (no storage form, a string
+		// that is not UTF-8, a repeated key, nesting too deep), never a
+		// value, so it is wrapped: errors.Is reads vcffi.ErrUnwritable.
+		return nil, fmt.Errorf("%w: the opaque value cannot be written as a storage blob: %w", encrypt.ErrEncoding, err)
 	}
 	return encoded, nil
 }
 
-// Opaque reads an opened opaque value into the generated shape struct: the
-// JSON document the engine returned, decoded into the exact Go types the
-// struct declares. Generated code calls it from Value.
+// Opaque reads an opened opaque value into the generated shape struct, at
+// the exact Go types the struct declares. Generated code calls it from
+// Value.
+//
+// The value is a storage blob, which begins "VCS". A value an earlier
+// release sealed is the JSON document encoding/json wrote, which begins
+// "{": it is read as JSON, so stored rows open without being encrypted
+// again. Each is read by its own rules: a storage blob gives back an empty
+// slice or map where the struct held nil, and JSON gives back nil.
 func Opaque[T any](vals Values, out *T) error {
 	v, ok := vals[OpaqueField]
 	if !ok {
@@ -403,6 +414,23 @@ func Opaque[T any](vals Values, out *T) error {
 	if !ok {
 		return fmt.Errorf("gensupport: %w: the opaque value opened as %T, not bytes", encrypt.ErrEncoding, v)
 	}
+	if bytes.HasPrefix(encoded, vcffi.StorageHeader[:3]) {
+		if err := vcffi.UnmarshalStorage(encoded, out); err != nil {
+			// The vcffi error names the kind found and where, never the
+			// value; it is wrapped so errors.Is reads its class
+			// (encrypt.ErrStorageVersion for a blob a later release
+			// wrote, say).
+			return fmt.Errorf("gensupport: %w: the opaque value does not read into a %T: %w", encrypt.ErrEncoding, *out, err)
+		}
+		return nil
+	}
+	// encoding/json writes a struct as an object, so a JSON document of
+	// the shape struct begins "{"; a value that begins with neither is not
+	// one either reader can take, and is said so rather than handed to
+	// the JSON reader to fail on.
+	if !bytes.HasPrefix(encoded, []byte("{")) {
+		return fmt.Errorf("gensupport: %w: the opaque value is neither a storage blob nor a JSON document", encrypt.ErrEncoding)
+	}
 	if err := json.Unmarshal(encoded, out); err != nil {
 		// Its error is not wrapped: encoding/json's text quotes the value.
 		return fmt.Errorf("gensupport: %w: the opaque value does not decode into a %T", encrypt.ErrEncoding, *out)

languages/golang/encrypt/gensupport/declaration.go

@@ -14,7 +14,7 @@ import (
 // derives; every field seals as vitaminc's tagged leaf whatever its kind,
 // which a Rust record opens when its field is a Value. Every sealed field has a
 // scalar kind: stashgen refuses a struct, slice or map outside an opaque
-// struct, and an opaque struct seals as Bytes (one JSON document). Untyped
+// struct, and an opaque struct seals as Bytes (one storage blob). Untyped
 // names a field with no declared type; generated code never writes it, and
 // the engine refuses a plan whose indexed field has no type.
 type Kind string
@@ -52,9 +52,10 @@ const (
 
 // OpaqueField is the one field of an opaque declaration: the whole struct,
 // sealed as one value under <context>/value. The struct crosses the binding
-// as one JSON document — the engine seals a composite value as a tree of
-// leaves, and an opaque struct is one column — so its field types are what
-// JSON carries: scalars, []byte (base64), slices and maps of them.
+// as one storage blob (vitaminc's storage profile, version 1) — the engine
+// seals a composite value as a tree of leaves, and an opaque struct is one
+// column. A value an earlier release sealed is a JSON document, which
+// [Opaque] still reads.
 const OpaqueField = "value"
 
 type verb uint8

languages/golang/encrypt/gensupport/gensupport.go

@@ -2,7 +2,7 @@
 //
 // A program never imports this package: it calls the functions the generated
 // file writes into its own package (users.Encrypt, users.Decrypt,
-// users.Fields). The generated file names [GeneratedVersion1], builds a
+// users.Fields). The generated file names [GeneratedVersion2], builds a
 // [Declaration] from the struct's tags, hands the library its conversions in
 // a [Generated] value, and prints through [Redacted] and [RedactedLog]. The
 // library lowers the declaration to the data plan the engine reads, sends a
@@ -25,9 +25,14 @@ import (
 // file names, so a file from another version does not compile.
 type generatedVersion uint8
 
-// GeneratedVersion1 is the layout of files that stashgen writes today. Every
-// generated file holds `const _ = gensupport.GeneratedVersion1`.
-const GeneratedVersion1 generatedVersion = 1
+// GeneratedVersion2 is the layout of files that stashgen writes today. Every
+// generated file holds `const _ = gensupport.GeneratedVersion2`.
+//
+// Version 2 seals an opaque struct as a storage blob keyed by its declared
+// field names (the shape struct's `vc` tags). A version 1 file has no such
+// tags, so it would write blobs keyed by Go field names that a regenerated
+// file cannot read; it does not compile against this library.
+const GeneratedVersion2 generatedVersion = 2
 
 // sealed is what a hidden field prints as.
 const sealed = "[sealed]"