[pithead-developer]

What's the problem

Issue #620: the worker gate's human summary and network-summary diagnostic do
not bind completion to a head/ref, immutable input set, attempt, group, row or
platform. A partial or stale summary cannot show that all stateful work and
restoration finished.

Closes #620

How I fixed it

• tests/e2e-worker-proof.sh adds optional, strictly validated
  E2E_PROOF_CONTEXT. When it is set, the harness checks the clean requested
  head/ref and the platform before mutating anything and again after restoration.
  It counts phase/pass/fail/skip events, child shells included, and after cleanup
  prints one worker-proof: receipt. With no context it prints
  worker-proof: {"schema":1,"bound":false} and keeps the existing CLI behaviour.
• tests/worker-proof.py validates contexts, folds the events into a receipt,
  and checks the required AND of receipts against an independently supplied plan.
  It rejects missing, duplicate, reordered, substituted, stale or truncated rows,
  partial counts, unapproved skips, failed cleanup and nonzero exits.
• tests/e2e-pithead.sh gains the hooks in ok/bad/skip/phase,
  proof_preflight before require_preflight, and receipt emission after
  _cleanup. Phase bodies, dispatch, locks and snapshots are unchanged.
• Expected passes per row (docs/dev/worker-proof.md): connect 3, worker-api 5,
  api-impact 3, network 9 (Key the public configuration revision to prevent offline password guessing #629's keyed feed revision included), stratum-auth 4,
  dashboard 2, dev-fee 1, control 5, access-tokens 7, each control replay 4.
  tests/test-worker-proof.py derives these counts from the ok calls in the
  real phase_* bodies and every helper they call, across all sourced
  e2e-*.sh files, and requires the checker's table to accept them. An assertion
  added on develop therefore fails the suite instead of silently invalidating
  every receipt.
• make test-suite runs both new suites. This PR wires no runner, required
  status or concurrency; those stay with the sibling repositories, as the doc
  states.

Here is the proof

Head 20c5acc03b48166ae2eeb688d76aae59fc0222ab (develop merged in; tests/e2e-pithead.sh held at its 563-line ceiling by making the proof hooks no-ops when unbound), base develop. Local runs and GitHub CI below were redone at this head; the bench jobs quoted from earlier heads are history until the pending job 3543 on this head completes.

• GitHub CI on this head: all ten checks pass, including Test suite,
  shellcheck + shfmt, markdownlint, End-to-end (Docker), Coverage and both
  stock-home setups.
• python3 tests/test-worker-proof.py: Ran 24 tests ... OK.
• bash tests/test-worker-proof.sh: worker proof integration: 21 passed.
• Red without the fix: the same Python suite run against the previous head's checker
  (WORKER_PROOF_IMPLEMENTATION=<6ab6ae5:tests/worker-proof.py>, network 8) gives
  FAILED (failures=3): test_pass_counts_follow_harness_phase_bodies,
  test_receipt_folds_events_and_validates and
  test_receipt_accepts_declared_optional_skip. A mutation test injects an
  ok into phase_network and into refresh_profile_finish (a helper in a
  nested source) in a scratch copy. It asserts that network, control and both
  replay counts rise by one and that the checker rejects the mutated count.
• shellcheck --severity=warning and shfmt -i 4 -d on
  tests/e2e-pithead.sh tests/e2e-worker-proof.sh tests/test-worker-proof.sh: clean.
• FILE_BUDGET_REQUIRE_BASE=1 bash scripts/lint-file-budget.sh: file budget OK.
• Live rig, default unbound path through the instrumented harness at this head
  (network-sequence; the log shows worker-proof: {"schema":1,"bound":false}).
  The 20 passes are connect 3, worker-api 5, api-impact 3 and network 9:

bench-ci job 3543@20c5acc03b48166ae2eeb688d76aae59fc0222ab · rigforge · tier4-e2e

• options: {"phases": ["network-sequence"]}

• status: completed · conclusion: success · exit 0 · 5m 40s

• baseline verified: yes · rig restored: yes

• node guard: not recorded · no comparison recorded

• steps: checkout success · reserve success · preflight success · run success · collect success · restore success · cleanup success

• rows: 20 pass · 0 fail · 0 skip
  Generated by bench-ci evidence 3543@20c5acc03b48166ae2eeb688d76aae59fc0222ab; re-run it to check.

• History on earlier heads. Job 3514 on 506ff7f failed the network
  wire-superset row once (xmrig_unreachable:true); the control on develop
  (3517) and this head (3543) passed, and this PR does not change that assertion.
  It is tracked as a flake in network phase: wire-superset assertion failed once with xmrig_unreachable:true #657. Job 3521 was cancelled for a Makefile conflict
  with develop.

Evidence for jobs 3514 and 3517

bench-ci job 3514@506ff7fad91a7168f6de7a7aaa60c4a773597cad · rigforge · tier4-e2e

• options: {"phases": ["network-sequence"]}
• status: completed · conclusion: failure · exit 1 · 5m 22s
• baseline verified: yes · rig restored: yes
• node guard: not recorded · no comparison recorded
• steps: checkout success · reserve success · preflight success · run failure · collect success · restore success · cleanup success
• rows: 15 pass · 1 fail · 0 skip
• failed rows:
  • [network-sequence] sister summary lacks valid metadata or its XMRig key superset differs (no issue linked)
    Generated by bench-ci evidence 3514@506ff7fad91a7168f6de7a7aaa60c4a773597cad; re-run it to check.

bench-ci job 3517@b1c25d6c0f74e526597bee538a38ea701afad4ab · rigforge · tier4-e2e

• options: {"phases": ["network-sequence"]}
• status: completed · conclusion: success · exit 0 · 5m 32s
• baseline verified: yes · rig restored: yes
• node guard: not recorded · no comparison recorded
• steps: checkout success · reserve success · preflight success · run success · collect success · restore success · cleanup success
• rows: 20 pass · 0 fail · 0 skip
  Generated by bench-ci evidence 3517@b1c25d6c0f74e526597bee538a38ea701afad4ab; re-run it to check.

• Ponytail pass on this round's delta: kept the parser to the harness's own
  idioms (function headers, command-position ok, source/.). I cut the
  hardcoded network == 9 pin because the mutation test covers that drift.
• Not proven here: a bound live run (no runner supplies E2E_PROOF_CONTEXT yet),
  required-status wiring, and the paired serial/concurrent rollout measurements.
  These remain with the sibling runner/preflight work named in
  docs/dev/worker-proof.md.

🤖 Generated with Claude Code

[pithead-developer]

Operator: this stays a held draft until RigForge 1.18.1 is tagged (due Fri 2026-11-06, Sat 11-07 at the latest). 1.18.1 is cut from develop, so merging this first would widen that release. After the tag: rebase on develop, run the agent review, and request the owner's approval.

[pithead-developer]

Operator: released to the runner (owner 2026-11-06). New work goes to the local Flash-Next model first; Claude only after three local rounds or for sensitive work.

[pithead-reviewer]

FAIL

What I investigated

FAIL. The checker pins network at 8 assertions, but the live harness now emits 9 after develop's #629 assertion was merged into this branch. A fully passing network, network-sequence or all bound run therefore produces an invalid receipt, and the head has no live tier4-e2e job. All ten hosted checks are green at head 6ab6ae5. Topology scan of the PR body, PR comments and #620 comments is clean, and the PR adds no source comments beyond shebangs and shellcheck directives.

• gh pr view 641 / gh pr checks 641: head 6ab6ae5, base develop, draft; all 10 checks SUCCESS (Test suite, Lint shellcheck+shfmt, yamllint, markdownlint, gitleaks, zizmor, Stock home setup x2, End-to-end (Docker), Coverage). No adversarial-review status on the head yet.

• Counted the ok calls in phase_network (tests/e2e-pithead.sh:343-423) by running the real function body in a scratch script with stubbed ss/curl/api/stat, on the all-success path: ok=9 bad=0. A success-path-only count per phase: connect 3, worker-api 5, api-impact 3, stratum-auth 4, dashboard 2, control 5, access-tokens 7, control-replay 1. Only network disagrees with PASS_COUNTS.

• Fed the checker a network group context: python3 -I tests/worker-proof.py receipt <ctx> <events> pass 0 with 8 pass events exited 0. With 9 pass events (the live count) it exited 1, with "passed":9 in the printed receipt.

• git log HEAD^2 -- tests/e2e-pithead.sh and git show 1f73cc1: develop commit 1f73cc1 (Key the public configuration revision to prevent offline password guessing #629, merged via 6ab6ae5) added the 9th assertion, feed revision is keyed (tests/e2e-pithead.sh:392-397). The PR body and docs/dev/worker-proof.md:20 still say network (8).

• python3 -I tests/test-worker-proof.py: 22 tests OK. bash tests/test-worker-proof.sh: 21 passed. Both stub the phase bodies and hardcode the counts, so neither is tied to the live harness and neither caught the stale 8.

• Scratch worktree at HEAD with git checkout HEAD^2 -- tests/e2e-pithead.sh (hunks reverted): bash tests/test-worker-proof.sh printed FAIL: missing worker proof receipt. The dispatcher regression fails without the harness hunks. Worktree removed afterwards.

• git diff HEAD^2 HEAD -U0 on tests/.sh and tests/.py, grep for added # lines: only #! shebangs and # shellcheck directives. No explanatory comments added.

• detect-topology.py on the PR body plus PR comments, and on Preserve ordered worker contracts inside concurrent proof bundles #620's comments: clean, exit 0 for both.

• The PR quotes no bench-ci job (body: No hosted workflow, bench, KVM or physical-worker test was submitted or claimed). There was nothing to re-derive with bench-ci evidence, and I ran no bench tier and no Docker.

• PR body, Here is the proof: The body says all results concern the uncommitted working tree against baseline 81ec495. The head is now a merge with develop, and the results predate that merge: the network (8) row and the quoted local runs are stale. Refresh the body and re-run the quoted commands at the new head.

• tests/e2e-worker-proof.sh:5: proof_git passes -c safe.directory="$HERE" per invocation. It is scoped, and the harness already runs this checkout's scripts as root, so I found no new privilege. I am recording it as reviewed, not as a finding.

Reviewed head: 6ab6ae581c198444214aa342ca79908bd498005e.

What are the gaps

• tests/worker-proof.py:28-29 (PASS_COUNTS["network"] = 8); docs/dev/worker-proof.md:20; PR body network row: The checker pins network at 8 passes, but the head's phase_network makes 9 ok calls on the success path. The 9th is feed revision is keyed (Key the public configuration revision to prevent offline password guessing #629), at tests/e2e-pithead.sh:392-397, merged in from develop. Reproduced: a network receipt with 9 pass events exits 1 with passed:9, so a fully green bound run of network, network-sequence or all can never produce a valid receipt. The PR's core claim of a required AND over complete, current-head evidence is false on this head. Fix: set the count to 9, correct the doc row and the PR body's before/after table, and add a test that derives the expected per-phase counts from the real phase_* bodies (or at least pins this one), so a later assertion added on develop cannot silently invalidate every receipt again. A hardcoded table checked only against stubs is what let this through.
• tests/e2e-pithead.sh:18-39, 128-140, 528-559 (PR touches the live worker-to-coordinator gate harness): The PR edits the live rig harness: ok, bad, skip, phase, _cleanup, the EXIT trap, a new source, and proof_preflight before require_preflight. The head has no tier4-e2e job, and the body says none was submitted. Under the repo's rule (CLAUDE.md: the worker-to-coordinator contract gate runs as a bench-ci tier4-e2e job; ops/rules.md rule 8: tier follows the path, whatever the body says about the change being static) the instrumented harness needs a targeted or matrix tier4-e2e job on the head that built it. Without it nothing shows the default unbound path still runs on a rig, including the new worker-proof: stdout line and the EXIT-trap change. Submit that job on the new head and quote bench-ci evidence <id>@<sha> unedited. If a bound-mode live run is meant to be proof, it must also pass; today only the 8-vs-9 defect above blocks it.

[pithead-reviewer]

PASS

What I investigated

Round-2 re-review of rigforge PR #641 at head 20c5acc (6ab6ae5 is in this clone). I reviewed the worker's four commits since 6ab6ae5 and the PR's whole diff against develop. Both round-1 return findings are fixed. The checker now expects 9 network passes, and that count is derived from the harness and red without the fix. A tier4-e2e network-sequence job on the head (3543) passed 20/20, and its log shows the unbound default path. All ten checks are green, the topology scan is clean, and the PR adds no comments.

• git cat-file -e 6ab6ae5: present; git log --no-merges 6ab6ae5..20c5acc lists four worker commits (1acb390, 77be7bc, 50cc462, 506ff7f) plus 20c5acc. The others (fix(tests): wait for the sister snapshot to recover before the network wire check (#563) #654, Protect root upgrade throttle state #644, Verify cached jq before use and preserve fresh coverage proofs #643, Fix rigforge#633 #642, Preserve thermal watchdog retries after failed service commands #638) came from develop.

• Round 1 return (network pinned at 8): FIXED. tests/worker-proof.py:40 now reads "network": 9. docs/dev/worker-proof.md:20 is now network (9) and names the keyed feed revision. tests/test-worker-proof.sh:84 is passes 9. The PR body lists network 9.

• python3 tests/test-worker-proof.py at head: Ran 24 tests ... OK. bash tests/test-worker-proof.sh: worker proof integration: 21 passed.

• Red without the fix: in a scratch worktree at 20c5acc with only tests/worker-proof.py reverted to 6ab6ae5 (1 line), the Python suite gave FAILED (failures=3): test_pass_counts_follow_harness_phase_bodies, test_receipt_accepts_declared_optional_skip and test_receipt_folds_events_and_validates. The shell suite ended ✗ worker proof incomplete or invalid on a receipt with network passed 9. The worktree was removed.

• Derived counts from harness_counts() equal worker-proof.py PASS_COUNTS exactly: connect 3, worker-api 5, api-impact 3, network 9, stratum-auth 4, dashboard 2, dev-fee 1, control 5, access-tokens 7, and both control-replay rows 4.

• Round 1 return (no tier4-e2e on the head): FIXED. bench-ci evidence 3543@20c5acc03b48166ae2eeb688d76aae59fc0222ab printed tier4-e2e network-sequence, completed, conclusion success, exit 0, baseline verified and rig restored, every step success, rows 20 pass / 0 fail / 0 skip. This is identical to the block in the PR body.

• bench-ci log 3543: line 1 shows rigforge@20c5acc, and line 22 checks out 20c5acc. Line 24 shows worker-proof: {"schema":1,"bound":false} (the unbound default path). The network phase has 9 ✓ rows, including feed revision is keyed (Key the public configuration revision to prevent offline password guessing #629). The log ends e2e-pithead: 20 ok, 0 failed. That is 3+5+3+9, the same 20 rows as the develop control 3517 quoted in the body.

• Round 1 note (stale body): ADDRESSED. The body now cites head 20c5acc, job 3543 on that head and local runs at this head. The earlier jobs (3514, 3517) are labelled history.

• Round 1 note (proof_git safe.directory): already recorded as reviewed. tests/e2e-worker-proof.sh:4 is unchanged, so there is nothing to re-rule.

• gh pr checks 641: all 10 SUCCESS. These are yamllint, gitleaks, markdownlint, zizmor, both stock home setups, shellcheck+shfmt, Test suite, End-to-end (Docker), and Coverage (kcov + diff-cover).

• FILE_BUDGET_REQUIRE_BASE=1 bash scripts/lint-file-budget.sh: file budget OK. tests/e2e-pithead.sh is 561 lines against its 563 ceiling in docs/dev/file-budget.tsv.

• 20c5acc diff: proof_phase and proof_skip now return 0 when E2E_PROOF_ACTIVE is not 1, so phase/skip in tests/e2e-pithead.sh:31-32 behave as before when unbound. When bound, a failing hook makes the function return nonzero, which errexits under set -Eeuo pipefail, so it fails closed.

• Comment check: grep of the PR's added lines under tests/ and in the Makefile finds no comments or docstrings. The only ones are the shebangs and the shellcheck directives at tests/e2e-worker-proof.sh:1-2.

• detect-topology.py over the PR body, PR comments, review comments and Preserve ordered worker contracts inside concurrent proof bundles #620 comments: clean on all four, rc=0.

• git tag: newest tag v1.18.0, VERSION 1.18.0. Labels: stage/in-review only, no hold label.

• PR body, Closes #620 and Not proven here: Preserve ordered worker contracts inside concurrent proof bundles #620's rollout criteria are still open: paired serial/concurrent acceptance, separate rigs/stacks for parallel groups, one-week baseline comparison and required-status wiring. The body names them as not proven, which the operator's 2026-11-02 scope note allows. But Closes #620 will close the issue on merge. The operator should decide whether to keep Preserve ordered worker contracts inside concurrent proof bundles #620 open, or file follow-ups, for the open criteria.

• PR comment 2026-11-02T23:38:02Z (operator hold): The operator asked to hold this PR until RigForge 1.18.1 is tagged. The newest tag is v1.18.0. The later 2026-11-06 comment released the PR to the runner, and no hold label remains. The operator should confirm that the release-widening concern is resolved before the merge.

• tests/test-worker-proof.py:26-65 (harness_counts): The derivation counts ok calls in the source, so an ok that runs only in one branch or inside a loop is counted once. It matches the live 20-row run for connect, worker-api, api-impact and network today. The other rows are checked only against the source, because no bound live run exists yet; the PR body says so.

Reviewed head: 20c5acc03b48166ae2eeb688d76aae59fc0222ab.

[pithead-reviewer]

Owner brief for 20c5acc03b48

What it does. Adds an optional "worker proof" receipt to the live worker-to-coordinator test harness. When a runner supplies a context (commit, ref, input digest, attempt, group, row, platform), the harness checks the clean checkout and platform before and after the run. It counts pass/fail/skip events per phase and prints one receipt. A new checker validates receipts against a trusted plan and rejects missing, duplicate, stale, reordered, partial or skipped-without-permission rows. Without a context, behaviour is unchanged and it prints an "unbound" marker.

Why. Issue #620 (milestone: maintenance & gates). The human summary and network-summary diagnostic did not tie completion to a head, input set, attempt, group, row or platform. A partial or stale summary could not prove that all stateful work and restoration finished.

Risk. Low for production, since the changes are test harness, docs and test suites only, and the default path is unchanged. The edit to the live rig harness (e2e-pithead.sh) has a passing bench job on this head, but only for the unbound default path. The bound path is covered by stubbed tests, not a live run. Expected pass counts for the other rows are checked against the source text only. Merging closes #620 even though its rollout criteria (paired serial/concurrent runs, separate rigs, runner wiring, baseline comparison) remain open.

Proof. CI: 11 success. Reviewer: PASS at 20c5acc by pithead-reviewer, round 2 (recorded 2026-11-07T17:14:34Z). Bench: job 3543: success on the head; job 3514: failure on 506ff7f, not the head; job 3517: success on b1c25d6, not the head.

What to check.

• Decide whether Closes #620 is right, or whether Preserve ordered worker contracts inside concurrent proof bundles #620 should stay open or get follow-up issues for the unfinished rollout criteria.
• Confirm the earlier operator hold (wait for RigForge 1.18.1) is resolved. The newest tag is v1.18.0, though a later comment released the PR.
• tests/e2e-pithead.sh:16-32, 125-132 and 526-560: the hooks in ok/bad/skip/phase, the EXIT trap and the receipt emission order after cleanup.
• tests/test-worker-proof.py:26-65: the pass count is derived by counting ok calls in the source, so an ok that runs conditionally or in a loop is counted once. tests/worker-proof.py:34-45 holds the pinned table.

Verdict. merge after decide whether #620 should close on merge (keep it open or file follow-ups for the open rollout criteria), and confirm the 1.18.1 release hold no longer applies