Bind ordered worker proofs to their execution context

Makefile

@@ -21,6 +21,8 @@ test-suite: ## rigforge test suite: unit + black-box, every CPU/OS profile simul
 	bash tests/test-docs-clone-pin.sh
 	bash tests/test-staged-rebuild.sh
 	bash tests/test-tool-pin-watch.sh
+	python3 tests/test-worker-proof.py
+	bash tests/test-worker-proof.sh
 	bash tests/test-coverage-cache.sh
 	bash tests/test-revision-key.sh
 

docs/dev/worker-proof.md

@@ -0,0 +1,133 @@
+# Ordered worker proof receipts
+
+`tests/e2e-pithead.sh` keeps each invocation on one reserved rig, in one shell,
+under one config/runtime snapshot. This change enables no concurrency and removes
+no per-PR or release witness. `network-summary` remains sanitized diagnostic
+information; it is not a completion receipt.
+
+## Stateful groups and retained assertions
+
+`all` runs the rows below in order. `network-sequence` runs the first four rows in
+that same order. Neither may be reconstructed by joining independently reset
+phases. Standalone phases remain available for diagnosis and existing row owners;
+they cannot substitute for either complete group.
+
+| Phase row | Retained successful assertions | Dependency and restoration |
+| --- | --- | --- |
+| `connect` (3) | Stratum job, accepted share, generated worker label | Sets the pool and restarts the physical worker. |
+| `worker-api` (5) | Open read, bind policy, restricted write refusal, unauthenticated refusal, matching bearer read | Uses the connected worker; restores the open token posture. |
+| `api-impact` (3) | Sister API available, loaded hashrate within baseline tolerance, responsiveness under mining load | Uses live hashing; starts/stops recorded load processes and disables the sister API. |
+| `network` (9) | Pool-only peers, control-implied listener, enabled API listener, miner-only listener policy, complete summary superset and metadata, keyed feed revision with a private key file, token leak sweep, pool-password leak sweep, disabled listener closed | Consumes the preceding mutations; restores disabled API/control and the open token posture. |
+| `stratum-auth` (4) | Right credential mines, wrong credential rejected, no jobs with wrong credential, recovery with right credential | Fault and recovery stay together; original config is restored by outer cleanup. |
+| `dashboard` (2) | Worker initially online, stopped worker becomes offline | Requires initial presence; stops then restarts the same worker. |
+| `dev-fee` (1) | Legacy minimum migration, zero donation, runtime reductions without rebuilding, original artifacts restored | The existing compound assertion includes real build/runtime and artifact restoration. |
+| `control` (5) | Apply accepted, donation applied, config persisted, appliance pool edit atomically refused, thermal change reaches a new feed within its deadline | Preserves systemd/timer profiling and the full observation window; restores profiling and outer snapshot. |
+| `access-tokens` (7) | Named token refused by miner, raw and derived reads accepted by sister API, apply accepted and applied, both revocation checks | Mutation and revocation stay together; outer cleanup restores tokens and donation. |
+
+Each replay group (`control-replay-thermal`, `control-replay-pools`) retains its
+runner-owned fixture/reservation, nested connect assertions (3), scalar apply
+sequence, pools mutation, optional thermal mutation, publication window and final
+assertion (1). Nested connect belongs to that replay row, not a separate group.
+Neither replay is added to or removed from `all`.
+
+Failures stop subsequent mutations. Cleanup still finishes refresh profiling,
+stops recorded load processes, restores and reapplies the saved config, compares
+its bytes, restores the original XMRig active state, and retains the snapshot if
+restoration fails. Receipt emission follows cleanup on normal and error exits.
+An interrupted or incomplete group cannot pass by presenting its successful prefix.
+
+## Binding an invocation
+
+The runner may supply `E2E_PROOF_CONTEXT` as one JSON object with exactly these
+fields. Do not put deployment coordinates, credentials, config bodies or private
+paths into this object.
+
+| Field | Meaning |
+| --- | --- |
+| `schema` | Integer `1`. |
+| `head` | Exact lowercase, forty-character Git commit ID. |
+| `ref` | Full Git ref identifying that commit. The ref must resolve locally. |
+| `input` | SHA-256 digest of the immutable, non-secret input manifest agreed by the runner. |
+| `run` | Fresh thirty-two-character lowercase hexadecimal attempt ID. |
+| `group` | Exact existing phase argument, `all`, or `network-sequence`. |
+| `row` | Runner-owned opaque lowercase row slug, unique within the required plan. |
+| `platform` | `linux-x86_64` or `linux-aarch64`, checked against the executing system. |
+| `allowed_skips` | Explicit map of permitted phase IDs to fixed skip reasons; normally `{}`. |
+
+Before acquiring the rig lock and after restoration, the producer checks the
+requested HEAD, ref resolution, platform, tracked-tree cleanliness, and absence of
+untracked executable inputs under the product/test directories. These calls mark
+the exact checkout path as trusted so root can read operator-owned checkouts.
+The input digest is a runner assertion: this script cannot establish the coordinator
+revision, image identities or fixture provenance. The runner must resolve and
+freeze those inputs before dispatch and retain the corresponding manifest.
+
+Without context the script emits `worker-proof: {"schema":1,"bound":false}` and
+retains the existing CLI behavior. This is explicitly unavailable bound evidence;
+the receipt checker rejects it. Supplying malformed context fails before rig
+mutation. This compatibility path does not implement the future required status.
+
+## Receipt and required AND
+
+A bound invocation emits the prefix `worker-proof:`, a space, and a JSON object:
+`schema`, `context`, `phases`, `cleanup`, and `exit_code`. Each phase has exactly
+`id`, `passed`, `failed`, and `skipped` (a list of fixed reason codes). The counts
+include assertions made in child shells. No assertion prose or response data is
+copied into the receipt. Counts are checked against the complete rows above;
+neither fewer nor extra assertions silently pass.
+`tests/test-worker-proof.py` derives each row's count from the `ok` calls in the
+harness phase bodies and the helpers they call, across every sourced `e2e-*.sh`
+file, so an assertion added to a phase fails the suite until the checker's table
+matches it.
+
+Only two existing fixture-absence skips are eligible for explicit authorization:
+
+- `stratum-auth: stratum-fixture-absent`, in `all` only. The standalone auth gate
+  still requires its fixture.
+- `dashboard: dashboard-fixture-absent`, in `all` or `dashboard`.
+
+A skipped row must contain exactly its authorized reason and no successful or
+failed assertions. A passing row remains valid when its skip was permitted but
+unused. Unknown or unexplained skips fail. An offline check after missing initial
+presence remains a failure and cannot be authorized as a skip.
+
+Extract each receipt as JSON into its own artifact and validate against an
+independently supplied plan, a nonempty JSON array of expected contexts:
+
+```bash
+python3 tests/worker-proof.py check PLAN.json RECEIPT.json MORE-RECEIPTS.json
+```
+
+The plan must retain every required row for one head/ref/input/attempt. Receipt
+arrival order may vary, but phase order within each group cannot. The checker
+rejects missing, duplicate, substituted or stale rows; wrong platform or context;
+missing, reordered, duplicated or unknown phases; partial assertion counts;
+unapproved skips; malformed/truncated data; nonzero exit; and failed cleanup.
+It must receive the trusted plan, never derive expectations from the artifacts it
+is checking. Receipts are not signed attestations: trust remains with the runner
+that owns the checkout, plan, fixtures and artifact collection.
+
+## Acceptance still required before rollout
+
+The sibling runner and evidence-preflight repositories must wire this required AND
+into the same current-head product/release status. Until then, local receipt
+validation alone does not establish the fleet gate. Keep all existing required
+statuses and per-PR proofs, including both independent bench-ci `make check` runs.
+The shared proof manifest and rank-7 immutable inputs remain coordinated work;
+review complete before/after behavior, assertion, input and platform inventories
+by hand until that manifest is available. Unknown or truncated evidence blocks
+acceptance, and existing row ownership remains authoritative.
+
+After rank-1 capacity acceptance, only independently proven groups may overlap,
+with distinct mutable rigs **and** stacks. Measure existing benches; obtain the
+owner's decision before adding a lane. Keep hashing/accepted-share, systemd/feed,
+donation, performance baselines, all `e2e-real` release phases, and every other
+currently required live/release witness.
+
+Before rollout, run hosted checks and paired live serial/concurrent acceptance on
+the same immutable inputs. Compare complete assertion/platform manifests, gate
+median/p90, occupied setup hours, failures and restoration. Record the baseline on
+normal authorized runs; one week after rollout compare paired PR/head latency,
+stage median/p90 minutes, occupied hours, failures and cleanup outcomes, reporting
+sample sizes and load. Roll back on lost evidence, cleanup failure or increased
+KVM queueing. None of those live measurements is claimed by the local regressions.

tests/README.md

@@ -7,6 +7,9 @@ compiles XMRig and mines. Each layer covers what the one below it has to stub.
 contract credential regression, plus access-token readiness, release clone-pin and offline tool-pin
 watcher regressions. The Bash suite exits nonzero if it aborts before its final
 `rigforge tests: ... passed, ... failed` summary, even when the shell would otherwise return zero.
+Worker proof schema/join and dispatcher regressions also run in `make test-suite`;
+see [ordered worker proof receipts](../docs/dev/worker-proof.md) for the retained
+phase inventory, binding contract and rollout requirements.
 The kcov container measures the Bash suite only; the credential regression always runs in the
 GitHub Test suite job, including argument inspection, password rotation and EXIT restoration.
 

tests/e2e-pithead.sh

@@ -14,20 +14,22 @@ RIGFORGE="$HERE/rigforge.sh"
 CFG="$HERE/config.json"
 source "$(dirname "${BASH_SOURCE[0]}")/e2e-pithead-control.sh" && source "$(dirname "${BASH_SOURCE[0]}")/e2e-pithead-tokens.sh" # phase_control (#509), phase_access_tokens (#516)
 
-source "$(dirname "${BASH_SOURCE[0]}")/e2e-control-replay.sh" && source "$HERE/tests/e2e-summary-diagnostics.sh"
+source "$(dirname "${BASH_SOURCE[0]}")/e2e-control-replay.sh" && source "$HERE/tests/e2e-summary-diagnostics.sh" && source "$HERE/tests/e2e-worker-proof.sh"
 # bench-ci-fixture: pithead-control control-replay-thermal control-replay-pools
 PASS=0 FAIL=0 E2E_EXIT_RC=0
 ok() {
+    [ "${E2E_PROOF_ACTIVE:-0}" != 1 ] || proof_assert pass
     PASS=$((PASS + 1))
     printf '  \033[1;32m✓\033[0m %s\n' "$1"
 }
 bad() {
+    [ "${E2E_PROOF_ACTIVE:-0}" != 1 ] || proof_assert fail
     FAIL=$((FAIL + 1))
     printf '  \033[1;31m✗\033[0m %s\n' "$1" >&2
     return 1
 }
-skip() { printf '  \033[1;33m∙\033[0m SKIP: %s\n' "$1"; }
-phase() { printf '\n\033[1m== e2e-pithead: %s ==\033[0m\n' "$1"; }
+skip() { proof_skip "$1" && printf '  \033[1;33m∙\033[0m SKIP: %s\n' "$1"; }
+phase() { proof_phase "${FUNCNAME[1]}" && printf '\n\033[1m== e2e-pithead: %s ==\033[0m\n' "$1"; }
 die() {
     printf '\033[31me2e-pithead: %s\033[0m\n' "$1" >&2
     exit 2
@@ -120,13 +122,14 @@ _cleanup() {
     }
     [ "$cleanup_ok" != 1 ] || rm -f "$SAVED_CFG" || cleanup_ok=0
     [ "${REPLAY_RUNNER_LOCK:-0}" = 1 ] || rm -f "${RIG_LOCK_HOLDER:-${RIG_LOCK_FILE:-/var/lock/rig-e2e.lock}.holder}" || true
+    [ "${E2E_PROOF_ACTIVE:-0}" != 1 ] || proof_cleanup "$cleanup_ok"
     [ "$cleanup_ok" = 1 ]
 }
 snapshot_config() {
     SAVED_CFG="$(mktemp)"
     cp "$CFG" "$SAVED_CFG"
     systemctl is-active --quiet xmrig 2>/dev/null && SAVED_XMRIG_ACTIVE=1 || SAVED_XMRIG_ACTIVE=0
-    trap 'E2E_EXIT_RC=$?; trap - EXIT; _cleanup || [ "$E2E_EXIT_RC" -ne 0 ] || E2E_EXIT_RC=1; exit "$E2E_EXIT_RC"' EXIT
+    trap 'E2E_EXIT_RC=$?; trap - EXIT; _cleanup || [ "$E2E_EXIT_RC" -ne 0 ] || E2E_EXIT_RC=1; if [ "${E2E_PROOF_ACTIVE:-0}" = 1 ]; then proof_finish "$E2E_EXIT_RC" || [ "$E2E_EXIT_RC" -ne 0 ] || E2E_EXIT_RC=1; fi; exit "$E2E_EXIT_RC"' EXIT
 }
 
 set_cfg() { # <jq program> [soft] [jq options...] -- soft: return 1 instead of dying when the edit or the apply fails (#514)
@@ -523,13 +526,15 @@ summary() {
     printf '0 failed\n'
 }
 
+proof_preflight "${1:-all}"
 require_preflight "$@"
 # Validate the phase against its own `phase_<name>` function: a typo dies with the rig untouched.
 # #183: the lock follows, before snapshot_config (whose _cleanup runs `apply`) and the first API touch.
 [ "${1:-all}" = all ] || declare -F "phase_${1//-/_}" >/dev/null || die "unknown phase '$1' (connect|worker-api|api-impact|network|network-sequence|stratum-auth|dashboard|dev-fee|control|control-replay-thermal|control-replay-pools|access-tokens|all)"
 replay_lock "${1:-all}"
 
 snapshot_config
+proof_begin
 case "${1:-all}" in
 connect) phase_connect ;;
 worker-api) phase_worker_api ;;
@@ -552,4 +557,5 @@ all)
 esac
 _cleanup || bad "pre-test config/runtime restoration failed; snapshot retained at $SAVED_CFG"
 trap - EXIT
+[ "${E2E_PROOF_ACTIVE:-0}" != 1 ] || proof_finish "$FAIL" || bad "worker proof incomplete or invalid"
 summary

tests/e2e-worker-proof.sh

@@ -0,0 +1,84 @@
+# shellcheck shell=bash
+# shellcheck disable=SC2034
+
+proof_git() { git -C "$HERE" -c safe.directory="$HERE" "$@"; }
+
+proof_identity() {
+    local head ref platform untracked
+    head=$(printf '%s' "$E2E_PROOF_CONTEXT" | jq -r .head) || return 1
+    ref=$(printf '%s' "$E2E_PROOF_CONTEXT" | jq -r .ref) || return 1
+    platform=$(printf '%s' "$E2E_PROOF_CONTEXT" | jq -r .platform) || return 1
+    untracked=$(proof_git ls-files --others --exclude-standard -- rigforge.sh util systemd tests) || return 1
+    [ "$(proof_git rev-parse --verify HEAD 2>/dev/null)" = "$head" ] &&
+        [ "$(proof_git rev-parse --verify "$ref^{commit}" 2>/dev/null)" = "$head" ] &&
+        [ "$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m)" = "$platform" ] &&
+        proof_git diff --quiet HEAD -- &&
+        [ -z "$untracked" ]
+}
+
+proof_preflight() {
+    E2E_PROOF_ACTIVE=0
+    [ -n "${E2E_PROOF_CONTEXT:-}" ] || {
+        printf 'worker-proof: {"schema":1,"bound":false}\n'
+        return 0
+    }
+    E2E_PROOF_CONTEXT=$(printf '%s' "$E2E_PROOF_CONTEXT" | python3 "$HERE/tests/worker-proof.py" context "$1") || die "invalid worker proof context"
+    proof_identity || die "worker proof requires the requested clean head and ref"
+}
+
+proof_begin() {
+    [ -n "${E2E_PROOF_CONTEXT:-}" ] || return 0
+    E2E_PROOF_ACTIVE=1 E2E_PROOF_CLEANUP=unknown
+    E2E_PROOF_DIR=$(mktemp -d) || return 1
+    printf '%s\n' "$E2E_PROOF_CONTEXT" >"$E2E_PROOF_DIR/context" || return 1
+    : >"$E2E_PROOF_DIR/events" || return 1
+    trap 'exit 130' INT
+    trap 'exit 143' TERM
+}
+
+proof_phase() {
+    [ "${E2E_PROOF_ACTIVE:-0}" = 1 ] || return 0
+    local id="${1#phase_}" group
+    id=${id//_/-}
+    group=$(printf '%s' "$E2E_PROOF_CONTEXT" | jq -r .group) || return 1
+    case "$group:$id" in
+    control-replay-*:connect) return 0 ;;
+    control-replay-*:control-replay) id=$group ;;
+    esac
+    printf '{"event":"phase","id":"%s"}\n' "$id" >>"$E2E_PROOF_DIR/events"
+}
+
+proof_assert() {
+    printf '{"event":"%s"}\n' "$1" >>"$E2E_PROOF_DIR/events"
+}
+
+proof_skip() {
+    [ "${E2E_PROOF_ACTIVE:-0}" = 1 ] || return 0
+    local reason=unexplained
+    case "$1" in
+    'E2E_STRATUM_PASS not set (stack auth off, or secret not provided) — phases skipped') reason=stratum-fixture-absent ;;
+    'E2E_DASH_URL not set — dashboard phases skipped (agree fixtures with pithead#209)') reason=dashboard-fixture-absent ;;
+    esac
+    printf '{"event":"skip","reason":"%s"}\n' "$reason" >>"$E2E_PROOF_DIR/events"
+}
+
+proof_cleanup() {
+    [ "$1" = 1 ] || E2E_PROOF_CLEANUP=fail
+    [ "${E2E_PROOF_CLEANUP:-unknown}" != unknown ] || E2E_PROOF_CLEANUP=pass
+}
+
+proof_finish() {
+    local rc="$1" result status=0
+    proof_identity || rc=1
+    if [ -n "${E2E_PROOF_DIR:-}" ]; then
+        result=$(python3 "$HERE/tests/worker-proof.py" receipt "$E2E_PROOF_DIR/context" "$E2E_PROOF_DIR/events" "${E2E_PROOF_CLEANUP:-fail}" "$rc") || status=1
+        if ! rm -rf "$E2E_PROOF_DIR"; then
+            result='' status=1
+        fi
+        [ -z "$result" ] || printf 'worker-proof: %s\n' "$result"
+    else
+        status=1
+    fi
+    E2E_PROOF_ACTIVE=0
+    return "$status"
+}