feat(stack-encrypt, eql-bindings, eql): EQL v4 order terms are typed, one hex string, checked when written

.changeset/eql-v4-order-term-shape.md

@@ -0,0 +1,5 @@
+---
+'@cipherstash/eql': minor
+---
+
+**An `eql_v4` order term is one hex string, and every `eql_v4` domain checks its shape when it is written.** A block ORE term (`ob`, and the `lo` and `hi` bounds of a block ORE range) is the lowercase hex of the whole ore-rs ciphertext, `"ob": "0202…"`, where it was a one-element array. Every `eql_v4` domain carrying `ob` must now hold a string starting `0202` or `0203`, and one carrying `op` a string starting `00` or `01`: an EQL v3 term or a term of the wrong scheme is refused with `check_violation` at the write or cast, not at the first comparison. The check reads only the prefix, so its cost does not grow with the term. In `eql-bindings`, a v4 `ob` field is Stack Encrypt's `OreBlockTerm` and an `op` field its `OpeTerm`, typed by the plaintext they order (`DateOrdOre::ob` is an `OreBlockTerm<Date>`), and reading a payload checks the term's structure. The `eql_v3` bundle is unchanged, byte for byte. A database that installed an earlier `eql_v4` bundle keeps its stored domains across an install, so it must drop its stored order domains first: see [U-003](../../docs/upgrading/v3.1.md#u-003-an-order-term-is-one-hex-string-checked-when-it-is-written).

Cargo.toml

@@ -120,6 +120,8 @@ rustls-platform-verifier = "0.6"
 rustls-webpki = { version = "0.103", default-features = false, features = ["alloc"] }
 serde = { version = "1.0", features = ["derive"] }
 serde_json = { version = "1.0.132" }
+# Constant-time hex (through base16ct) for serde over the order terms.
+serdect = { version = "0.3", default-features = false, features = ["alloc"] }
 thiserror = "1.0.56"
 tokio = { version = "1.47.1", features = ["full"] }
 tracing = { version = "0.1", features = ["log"] }

languages/golang/CHANGELOG.md

@@ -187,6 +187,13 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
 
 ### Changed
 
+- **An EQL v4 block ORE term is one hex string.** The `ob` of a value
+  encrypted into an `_ord_ore` or `text_search_ore` EQL type is
+  `"ob": "0202…"`, where it was a one-element array, and the EQL v4 bundle
+  checks the shape of every `ob` and `op` it stores. A database that
+  installed the EQL v4 bundle attached to `v0.1.0-alpha.1` must drop its
+  stored order domains before installing the new one (EQL's upgrade note
+  U-003).
 - **A call whose context ends fails on its own and leaves the client
   open.** A cancelled or timed-out call used to close the `encrypt.Client`,
   or the `auth` profile behind its strategy, for every later caller, and a

packages/eql/crates/eql-bindings/CHANGELOG.md

@@ -20,6 +20,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- **A v4 order term is Stack Encrypt's own, typed by its plaintext**
+  (`stack-encrypt` feature). An `ob` field is
+  `stack_encrypt::sem::OreBlockTerm<T>` and an `op` field
+  `stack_encrypt::sem::OpeTerm<T>`, over the type's plaintext:
+  `DateOrdOre::ob` is an `OreBlockTerm<Date>`, `IntegerOrd::op` an
+  `OpeTerm<i32>`. Each is one lowercase hex string on the wire (`ob` was a
+  one-element array) and is decoded through `from_bytes` when a payload is
+  read, so an EQL v3 term, a term of the other scheme or a malformed one is
+  refused there. `v4::terms` no longer re-exports v3's `OreBlock256` and
+  `OpeCllw`; the generated `v4::terms::OreBlockHex` and `OpeHex` describe
+  the JSON to TypeScript and JSON Schema, with the pattern the domain CHECKs
+  require.
+
 - **`v4::plaintext::Date`, `Timestamp`, `Decimal` and `RangeError` are
   Stack Encrypt's** (`stack_encrypt::value`), re-exported at the same
   path: the engine orders them, which it can only do for types of its own.
@@ -33,7 +46,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   on for `smallint`, `bigint`, `date`, `timestamp`, `numeric`, `real` and
   `double`, listed by `v4::all_range_query()`. Each holds both bounds of one
   `BETWEEN`: each bound is the single-bound domain's `ob` (ORE,
-  `OreBlock256`) or `op` (OPE, `OpeCllw`) value, in a `{v, i, lo, hi}`
+  `OreBlockTerm`) or `op` (OPE, `OpeTerm`) value, in a `{v, i, lo, hi}`
   envelope, so a range binds as one parameter. The parameter must be cast to
   its exact range domain (`$1::eql_v4.query_integer_range_ore`); anything else
   fails with SQLSTATE 42725. They bind the

packages/eql/crates/eql-bindings/README.md

@@ -38,6 +38,14 @@ Shared wire fields are reusable newtypes in
 | `OreBlock256` | `ob` | `Vec<String>` | `_ord_ore` domains, `text_search_ore` |
 | `BloomFilter` | `bf` | `Vec<i16>` (signed!) | `_match` domains |
 
+EQL v4 shares these except its order terms. A v4 `ob` field is Stack
+Encrypt's `stack_encrypt::sem::OreBlockTerm<T>` and an `op` field its
+`OpeTerm<T>`, over the type's plaintext (`v4::date::DateOrdOre::ob` is an
+`OreBlockTerm<Date>`). Each is one lowercase hex string on the wire, and
+reading a payload checks the term's structure, so an EQL v3 term or a term
+of the other scheme is refused. `v4::terms::OreBlockHex` and `OpeHex`
+describe that JSON to TypeScript and JSON Schema.
+
 Note "v3" names the SQL schema generation (`eql_v3.*`); the JSON envelope
 version is `v: 3` — the generated domain CHECKs assert `VALUE->>'v' = '3'`, and
 the wire field names are unchanged from v2 (the purpose-named rename in
@@ -133,8 +141,8 @@ and `rust_decimal` features convert to and from `NaiveDate`,
 caller's, and is not wiped. Only a leap second, which a `DateTime<Utc>` can
 hold and a `Timestamp` cannot, is refused in a conversion. On `NumericEq`,
 `1.5 = 1.50` (the scale is normalised); on `TimestampEq`, an instant is truncated to the microsecond (the nanoseconds below it are dropped, not rounded), so two instants in the same microsecond are equal. Each opens as
-it was stored. The ordered date, timestamp and numeric types are refused
-until the engine derives their order terms.
+it was stored. Their ordered types order a date by its day, a timestamp by
+the microsecond (truncated, as on `TimestampEq`) and a decimal by value.
 
 Each seals the same leaf the Go SDK seals for the same value, so values
 written here and by the Go SDK open each other. On the equality

packages/eql/crates/eql-bindings/src/encryption.rs

@@ -100,9 +100,7 @@ pub mod targets;
 use base64::{engine::general_purpose::STANDARD, Engine as _};
 #[allow(deprecated)]
 use stack_encrypt::sem::MatchTerm;
-use stack_encrypt::sem::{
-    EqualityTerm, MatchConfig, MatchOptions, OpeTerm, Orderable, OreBlockTerm, TermError, Tokenizer,
-};
+use stack_encrypt::sem::{EqualityTerm, MatchConfig, MatchOptions, TermError, Tokenizer};
 use stack_encrypt::target::transcode::{Reader, Transcode, Visitor};
 use stack_encrypt::target::{self, AeadContext, CallerContext, IndexSpec};
 use stack_encrypt::{
@@ -112,7 +110,7 @@ use stack_encrypt::{
 };
 
 use crate::{
-    v3::terms::{BloomFilter, Ciphertext, Hmac256, OpeCllw, OreBlock256},
+    v3::terms::{BloomFilter, Ciphertext, Hmac256},
     v4::terms::SearchBloomFilter,
     Identifier,
 };
@@ -450,97 +448,6 @@ impl<P, Ctx> DecryptField<P, Ctx> for SearchBloomFilter {
     }
 }
 
-/// Builds EQL's hex OPE term from the native order-preserving term.
-///
-/// The bytes are Stack Encrypt's CLLW OPE ciphertext of the value's
-/// ADR-0002 order key, which `vitaminc-ore` encodes: an integer at its own
-/// width, a float as the `f64` it widens to (`-0.0` ties `0.0`, every NaN is
-/// one value after `+Infinity`), text in Unicode 16 NFC with case and accents
-/// folded. EQL compares `op` as `bytea`, which is the lexicographic byte
-/// order the scheme preserves, so `<` on the stored terms is `<` on the
-/// order keys. Matching cipherstash-client's bytes is not a goal:
-/// cipherstash-client never writes an `eql_v4` column.
-///
-/// Text with no fold (a code point Unicode 16 does not assign) has no order
-/// term and is refused, stored or queried.
-#[doc(hidden)]
-pub struct OpeVisitor;
-impl Visitor for OpeVisitor {
-    type Value = OpeCllw;
-    fn ope<T: Orderable>(self, term: OpeTerm<T>) -> Result<Self::Value, Error> {
-        Ok(OpeCllw(hex::encode(term.as_bytes())))
-    }
-}
-impl Transcode for OpeCllw {
-    type Visitor = OpeVisitor;
-    fn visitor() -> Self::Visitor {
-        OpeVisitor
-    }
-}
-impl<S: Orderable + Clone> EncryptFrom<S> for OpeCllw {
-    type Context = CallerContext;
-    fn encryption<'s, K: 'static>() -> Encryption<'s, S, Self, K, Self::Context>
-    where
-        S: 's,
-    {
-        target::ope().transcode()
-    }
-}
-impl Decryptable for OpeCllw {
-    const DECRYPTABLE: bool = false;
-}
-impl<P, Ctx> DecryptField<P, Ctx> for OpeCllw {
-    fn decryption_field<K: 'static>(self, _: Ctx) -> Option<Decryption<P, K>> {
-        None
-    }
-}
-
-/// Builds EQL v4's `ob` from the native block ORE term: one element, the
-/// hex of the whole ore-rs ciphertext.
-///
-/// The ciphertext is ore-rs's 6-bit block ORE of the value's ADR-0002 order
-/// key (`OreBlockTerm`): the fixed-width scheme for integers and floats, the
-/// chained one, padded to a power-of-two length bucket of at least 16
-/// bytes, for text. EQL v4 compares `ob` with the comparator for that
-/// format, which takes exactly one element; EQL v3's `ob` holds 8-bit block
-/// terms, up to six for text, that cipherstash-client writes. A query
-/// carries the full ciphertext too: the comparator reads the right half of
-/// the operand a btree index or `between` passes second.
-///
-/// Text with no fold, text whose fold ends in NUL or is longer than 1023
-/// bytes has no block ORE term and is refused, stored or queried.
-#[doc(hidden)]
-pub struct OreBlockVisitor;
-impl Visitor for OreBlockVisitor {
-    type Value = OreBlock256;
-    fn ore_block<T: Orderable>(self, term: OreBlockTerm<T>) -> Result<Self::Value, Error> {
-        Ok(OreBlock256(vec![hex::encode(term.as_bytes())]))
-    }
-}
-impl Transcode for OreBlock256 {
-    type Visitor = OreBlockVisitor;
-    fn visitor() -> Self::Visitor {
-        OreBlockVisitor
-    }
-}
-impl<S: Orderable + Clone> EncryptFrom<S> for OreBlock256 {
-    type Context = CallerContext;
-    fn encryption<'s, K: 'static>() -> Encryption<'s, S, Self, K, Self::Context>
-    where
-        S: 's,
-    {
-        target::ore_block().transcode()
-    }
-}
-impl Decryptable for OreBlock256 {
-    const DECRYPTABLE: bool = false;
-}
-impl<P, Ctx> DecryptField<P, Ctx> for OreBlock256 {
-    fn decryption_field<K: 'static>(self, _: Ctx) -> Option<Decryption<P, K>> {
-        None
-    }
-}
-
 #[cfg(test)]
 mod tests {
     use stack_encrypt::{nonempty, Descriptor, Label};

packages/eql/crates/eql-bindings/src/v4/bigint.rs

@@ -1,10 +1,11 @@
 // @generated by eql-codegen from the eql-domains catalog — do not edit
 //! The `bigint` encrypted-domain family — generated from the eql-domains catalog.
-use crate::v4::terms::{Ciphertext, Hmac256, OpeCllw, OreBlock256};
+use crate::v4::terms::{Ciphertext, Hmac256, OpeHex, OreBlockHex};
 use crate::v4::{DomainType, SchemaVersion};
 use crate::Identifier;
 use schemars::{schema_for, JsonSchema, Schema};
 use serde::{Deserialize, Serialize};
+use stack_encrypt::sem::{OpeTerm, OreBlockTerm};
 use ts_rs::TS;
 /// `public.eql_v4_bigint` — storage-only domain.
 ///
@@ -129,7 +130,9 @@ pub struct BigintOrdOre {
     #[stash(context_field)]
     pub i: Identifier,
     pub c: Ciphertext,
-    pub ob: OreBlock256,
+    #[ts(as = "OreBlockHex")]
+    #[schemars(with = "OreBlockHex")]
+    pub ob: OreBlockTerm<i64>,
 }
 impl DomainType for BigintOrdOre {
     fn sql_domain_static() -> &'static str {
@@ -179,7 +182,9 @@ pub struct BigintOrd {
     #[stash(context_field)]
     pub i: Identifier,
     pub c: Ciphertext,
-    pub op: OpeCllw,
+    #[ts(as = "OpeHex")]
+    #[schemars(with = "OpeHex")]
+    pub op: OpeTerm<i64>,
 }
 impl DomainType for BigintOrd {
     fn sql_domain_static() -> &'static str {
@@ -229,7 +234,9 @@ pub struct BigintOrdOpe {
     #[stash(context_field)]
     pub i: Identifier,
     pub c: Ciphertext,
-    pub op: OpeCllw,
+    #[ts(as = "OpeHex")]
+    #[schemars(with = "OpeHex")]
+    pub op: OpeTerm<i64>,
 }
 impl DomainType for BigintOrdOpe {
     fn sql_domain_static() -> &'static str {
@@ -328,7 +335,9 @@ pub struct BigintOrdOreQuery {
     pub v: SchemaVersion,
     #[stash(context_field)]
     pub i: Identifier,
-    pub ob: OreBlock256,
+    #[ts(as = "OreBlockHex")]
+    #[schemars(with = "OreBlockHex")]
+    pub ob: OreBlockTerm<i64>,
 }
 impl DomainType for BigintOrdOreQuery {
     fn sql_domain_static() -> &'static str {
@@ -377,7 +386,9 @@ pub struct BigintOrdQuery {
     pub v: SchemaVersion,
     #[stash(context_field)]
     pub i: Identifier,
-    pub op: OpeCllw,
+    #[ts(as = "OpeHex")]
+    #[schemars(with = "OpeHex")]
+    pub op: OpeTerm<i64>,
 }
 impl DomainType for BigintOrdQuery {
     fn sql_domain_static() -> &'static str {
@@ -426,7 +437,9 @@ pub struct BigintOrdOpeQuery {
     pub v: SchemaVersion,
     #[stash(context_field)]
     pub i: Identifier,
-    pub op: OpeCllw,
+    #[ts(as = "OpeHex")]
+    #[schemars(with = "OpeHex")]
+    pub op: OpeTerm<i64>,
 }
 impl DomainType for BigintOrdOpeQuery {
     fn sql_domain_static() -> &'static str {
@@ -467,8 +480,12 @@ impl DomainType for BigintOrdOpeQuery {
 pub struct BigintRangeOreQuery {
     pub v: SchemaVersion,
     pub i: Identifier,
-    pub lo: OreBlock256,
-    pub hi: OreBlock256,
+    #[ts(as = "OreBlockHex")]
+    #[schemars(with = "OreBlockHex")]
+    pub lo: OreBlockTerm<i64>,
+    #[ts(as = "OreBlockHex")]
+    #[schemars(with = "OreBlockHex")]
+    pub hi: OreBlockTerm<i64>,
 }
 impl DomainType for BigintRangeOreQuery {
     fn sql_domain_static() -> &'static str {
@@ -509,8 +526,12 @@ impl DomainType for BigintRangeOreQuery {
 pub struct BigintRangeOpeQuery {
     pub v: SchemaVersion,
     pub i: Identifier,
-    pub lo: OpeCllw,
-    pub hi: OpeCllw,
+    #[ts(as = "OpeHex")]
+    #[schemars(with = "OpeHex")]
+    pub lo: OpeTerm<i64>,
+    #[ts(as = "OpeHex")]
+    #[schemars(with = "OpeHex")]
+    pub hi: OpeTerm<i64>,
 }
 impl DomainType for BigintRangeOpeQuery {
     fn sql_domain_static() -> &'static str {