Repository navigation
Description
Problem
The merge scripts judge a saved check rollup with the shell function mp_judge_checks in packages/agents/source/skills/merge-pr/shared/scripts/pull-request.sh, using jq. internal/checks holds a Go evaluator, which the unattended merge path calls through agentrepo checks evaluate (ma_gates in merge-agent.sh). Moving the attended path to the Go evaluator is feature 3 of #588. It is unsafe to do before the two judges' differences are written down, because a difference can turn a failure green or a green into a refusal. The Feature that does the move is #723, and it is blocked by this Spike.
The two judges differ on the same data (read on main at d41369e7):
- Malformed input: the shell exits 3, Go exits 5. A missing or null
statusCheckRollup: the shell reads it as an empty rollup (exit 4), Go as unreadable input. - Unknown states: the shell reads an unknown conclusion, status or status-context state as ok. Go returns the failing outcome (exit 1) for them, not the unreadable outcome (
internal/checks/checks.go:180,internal/checks/checks_test.go:82). A missingstatusreads as completed in the shell and as failing in Go. The shell upper-cases states before comparing, so a lowercase spelling reads like its upper-case form; Go compares exactly and treats a lowercase spelling as failing. - Superseded entries: both judges select the latest run of each check before reading states (
internal/checks/checks.go:148,pull-request.sh:88), so an unknown state on an earlier run that a later success replaced is never read. The contract has to say whether a refusal for unknown states covers such runs. - Entries without a name or with an unknown
__typename: the shell judges them, Go rejects them as unreadable. - Start times: the shell compares
startedAtas strings (pull-request.sh:71,pull-request.sh:88), Go parses instants (internal/checks/checks.go:110,internal/checks/checks.go:269). A failing run at2026-10-01T02:00:00+02:00and a passing run at2026-10-01T01:00:00Zof the same check make the shell choose the failure and Go choose the success. An unparseable timestamp is unreadable input in Go and a string in the shell. Tied start times: the shell takes whichevermax_byreturns, Go takes the worse verdict. - Repeated status contexts: the shell judges every entry as it stands, so an older failing entry fails the rollup. Go keeps only the newest entry per context, so a newer success hides it.
The callers also depend on the shell judge's behaviour. preflight.sh calls mp_require_reviews first and exits 75 when a review is required (preflight.sh:94; the regression at tests/skills/merge-pr/scripts/merge-workflow.bats:385 requires it), runs the unattended gates, and only then reports each check's state and name, sorted by name, without failing or still-running checks making that section refuse. squash-merge.sh prints failing or still-running checks as <state> <name> lines on standard error and refuses with a message that depends on the outcome. Both callers parse the saved response before they judge it (preflight.sh:78, squash-merge.sh:116, pull-request.sh:39), so a truncated live response can stop them with a parser error before any judge message. The Go command prints a JSON summary of the non-successful names only, with exit codes 0 to 6 (internal/agentrepo/command.go:55). The shared loader accepts agentrepo from v0.16.0 (packages/agents/source/lib/lib.sh:140), and docs/agent-skills-migration.md:9 lets a consumer run with none.
Acceptance criteria
- AC1: The findings comment holds a contract table with one row for each of these input classes: malformed JSON, missing or null rollup, empty rollup, unknown state, lowercase state, missing status, empty check name, unknown entry type, tied start times, a rerun that has not started, repeated status contexts, start times in different UTC offsets, an unparseable start time, and an unknown state on a superseded run. Each row states the shell judge's outcome and exit status, the chosen preflight and squash outcomes separately, and whether the change is deliberate. For every case, each caller's chosen result records its report/continuation or refusal, numeric process exit status, and exact standard output and standard error (including the diagnostic message and explicitly empty streams). These are caller process statuses, separate from the judge's status; otherwise passing prerequisites and later gates isolate the case, with merge writes stubbed. Invocation modes and earlier stops are identified as in AC7. Empty, missing and null rollups have separate cases. These fields specify findings for Replace the shell check judge in the merge scripts with the Go evaluator #723 to consume; they do not require implementing the chosen behavior in this Spike.
- AC2: Every row of the table cites at least one saved rollup, with the output and exit status the shell judge produced for it and the result the current Go evaluator gives for it, both recorded by running them on the head named in the comment.
- AC3: The comment states the sweep's domains, generation rule and expected total before reporting results. The sweep uses homogeneous ordered pairs of two entries for one check, separately for
CheckRun(ordered bystartedAt) andStatusContext(ordered bycreatedAt); it does not pair different entry types. For each CheckRun, thestatusdomain isCOMPLETED,IN_PROGRESS,PENDING,QUEUED,REQUESTED,WAITING, all six exact lowercase spellings, the literalUNKNOWN_STATE, and omission ofstatus(14 alternatives). Its independentconclusiondomain isACTION_REQUIRED,CANCELLED,FAILURE,NEUTRAL,SKIPPED,STALE,STARTUP_FAILURE,SUCCESS,TIMED_OUT, all nine exact lowercase spellings, the literalUNKNOWN_STATE, and omission ofconclusion(20 alternatives). Cross the full status × conclusion product on each entry, including combinations in which status causes a judge to ignore conclusion; do not vary one field at a time. For each StatusContext, thestatedomain isERROR,EXPECTED,FAILURE,PENDING,SUCCESS, all five exact lowercase spellings, the literalUNKNOWN_STATE, and omission ofstate(12 alternatives). Omission means the property is absent, not null or an empty string; neither null nor empty-string state fields nor a lowercase unknown sentinel are additional sweep alternatives. Each ordered state pair takes all seven concrete timestamp pairs defined in Approach. Other fields are fixed as stated there. The expected total is7 × ((14 × 20)^2 + 12^2) = 549808cases: 548800 CheckRun pairs and 1008 StatusContext pairs, before any deduplication of equivalent judge results. The recorded cases cover exactly that product, and zero combinations that the shell judge reports as failing or still running are green under the table's outcomes. - AC4: The comment states one general decision rule for any pair of runs of one check in which the string order and the instant order of the start times disagree. The rule is never green when either ordering would select a failing or unfinished run, and the table's row for the failing run at
2026-10-01T02:00:00+02:00against the passing run at2026-10-01T01:00:00Zapplies it. - AC5: For an unknown status, conclusion or status-context state, on the latest run and on a superseded run, the table records the chosen preflight and squash outcomes separately, each with its numeric process exit status and exact standard output and standard error, including its message or explicitly empty streams, using AC1's caller fields and boundary. It states whether validation covers every entry or only selected entries.
- AC6: A second table covers every exit status of the evaluator command (0 to 6), any other exit status, an absent command, malformed or non-result output, valid-looking success output with a nonzero exit status (including mapped statuses 1 and 2), and a zero exit status with missing result fields. For every case it records the evaluator's process status and output (or absence), the chosen evaluation category (green, failing, still running, empty or unreadable), and separate chosen preflight and squash results using all AC1 caller fields: report/continuation or refusal, numeric process exit status, exact standard output and standard error with the message or explicitly empty streams. Only a zero evaluator exit status with a complete success result is green; preflight continuation is not evidence of green checks. The table identifies whether an earlier stop or the check section supplies each result.
- AC7: A third table lists each refusal and parser stop before the shell check-reporting and judging section, including those raised by the shared loader (for example its version refusal) and by the unattended helpers it invokes (command usability and the unattended gates). For every case and relevant invocation mode it records preflight and squash results separately using AC1's caller fields: report/continuation or refusal, numeric process exit status, exact standard output and standard error with the message or explicitly empty streams, the first stop reached and whether the check section was reached. It states whether each result stays unchanged. Record the other caller's result even when that caller does not take the same stop. It states that preflight's check section reports failing and still-running checks without refusing, and that a required review still exits 75.
- AC8: The comment names the unattended path's existing fixtures that the table changes and the outcome, exit status, messages and skipped-check hint each gets under the table, or states that none change.
- AC9: The comment lists the evaluator outputs the table promises that the current evaluator does not yet give, and either names the first devtools release that gives all of them or states that no compatible release exists yet and names the issue that implements and releases it. In both cases it states how the callers behave with an older release or with none, and names the issue that installs a compatible release in consumer repositories.
- AC10: Non-functional: the comment records the median time of 20 runs of the shell judge and of the Go evaluator over a generated 500-check rollup, and proposes the time limit for the merge path.
- AC11: Each AC of this issue is answered in the findings comment, and the comment links the Feature issue for the caller migration and any further issue filed.
- AC12: Non-functional: before posting, the complete final findings comment and every complete final follow-up issue body, including embedded corpus and command output, each pass gitleaks, the repository's
check-leaks.shwith its private-strings list, and a separate leak review of each post's full text. The findings comment records a clean result for each of the two scans and the leak review for each post. Any text changed after a check is checked again before posting.
Scope and non-goals
In scope: reading and running the two judges, recording the three tables, the corpus and the measurements, and filing follow-up issues.
Non-goal: changing the shell judge, the Go evaluator, the merge scripts or the generated skills; the findings are a comment, and any prototype stays on a throwaway branch that is deleted.
Non-goal: review-verdict parsing (#589), approval selection (#590), policy inspection and merge execution (later features of #588).
Out of scope: distributing the Go binary (#333) and installing it in consumer repositories (#544).
Approach
Build the evaluator command from main, then run the shell judge and the Go evaluator over hand-written rollups for each class of AC1 and over the generated sweep of AC3. Record every rollup with both outputs, so the Feature can commit them as fixtures. Choose each row's outcome by one rule, with no exceptions: the merge path is never greener than the shell judge for a failing or unfinished check, and a rollup that cannot be read is refused. Before running the sweep, write down its domains, generation rule and expected total (AC3). The seven ordered timestamp pairs (first, second) are: equal (2026-10-01T01:00:00Z, 2026-10-01T01:00:00Z); earlier (2026-10-01T00:00:00Z, 2026-10-01T01:00:00Z); later (the preceding pair reversed); offset-reversed (2026-10-01T02:00:00+02:00, 2026-10-01T01:00:00Z); unparseable first (INVALID_TIME, 2026-10-01T01:00:00Z); unparseable second (the preceding pair reversed); and both unparseable (INVALID_TIME, INVALID_TIME). Apply these to startedAt for CheckRuns and createdAt for StatusContexts. Keep JSON array order first then second, __typename fixed to that sub-sweep's type, one fixed nonempty check name and workflow for CheckRuns and one fixed nonempty context for StatusContexts; omit fields belonging only to the other type. Fix the head and use no required-check names or expected-head constraint for this comparative sweep; AC8 separately records unattended requirements. No other axis is multiplied into AC3; AC1's additional cases remain separate.
Trace the shared loader and the unattended helpers invoked before the check section, as well as the three callers, to fill the third table, and read the unattended fixtures under tests/skills/agent-merge-pr/ for AC8. Post the findings as one comment on this issue.
Dependencies
Native parent: #588. No blocker. The Feature that migrates the callers, #723, is blocked by this issue, by #333 and by #544.
Size
S. One findings comment and, at most, a few follow-up issues.
Verify plan
- AC1: the table has a row for every class named in AC1, with distinct empty, missing and null rollup cases. Inspect every row's separate chosen preflight and squash report/continuation or refusal, numeric process exit status, exact standard output and standard error (message or explicitly empty streams), invocation mode and boundary, rather than accepting only a merge-path category or evaluator exit code.
- AC2: each row cites a rollup and both recorded outputs; a reader can rerun the two commands on the cited rollup and get the recorded result.
- AC3: independently reconstruct the specified field-level products: check all 14 status alternatives, all 20 conclusion alternatives and all 280 crossed CheckRun entries, including simultaneous lowercase/missing fields and combinations with non-completed status; check all 12 context-state alternatives. Compare the generated case keys with the complete ordered-pair × seven-timestamp set, requiring no missing or duplicate keys and exactly 548800 CheckRun and 1008 StatusContext cases (549808 total). A match between a researcher-selected total and a recorded total alone does not pass. Inspect the domain manifest and the seven timestamp pairs, then confirm zero shell-failing or still-running combinations become green under the chosen outcomes.
- AC4: the stated general rule, the row for the two runs above, and the sweep's offset-reversed combinations (AC3), none of which shows a green outcome when either ordering selects a failing or unfinished run.
- AC5: inspect unknown status, conclusion and context-state rows on both latest and superseded entries; require the validation-scope decision and separate chosen preflight and squash results, each with numeric process exit status and exact standard output and standard error, including its message or explicitly empty streams.
- AC6: inspect every named case in the second table, including malformed evaluator output, success-looking output followed by exits 1 and 2, and exit 0 with missing result fields. Check the recorded evaluator status/output separately from both callers' chosen report/continuation or refusal, numeric process exit status and exact standard output and standard error (message or explicitly empty streams); confirm only a complete success result with evaluator exit 0 is green. Confirm the current evaluator's statuses against
internal/agentrepo/command.goandinternal/checks/checks.goon the cited head, and record the chosen caller contract as research findings. - AC7: confirm the third table against
preflight.sh,squash-merge.sh,pull-request.sh, the shared loader andmerge-agent.shon the cited head, tracing the helpers invoked before the check section for each invocation mode, and the regression attests/skills/merge-pr/scripts/merge-workflow.bats:385. Inspect both callers' numeric process exit statuses, exact standard output and standard error (including messages or explicitly empty streams), first stop, reachability of the check section and unchanged/change decision for every case; do not substitute a judge exit status for a caller status. - AC8: the list of unattended fixtures, each run on the cited head under the proposed rules.
- AC9: the list of promised outputs the current evaluator lacks, then either the named release with evidence that it produces them or the "no compatible release yet" statement with the implementing issue, and the issue number that installs it.
- AC10: the recorded medians and the commands that produced them.
- AC11: the comment's answer to each AC and the links.
- AC12: before posting, run gitleaks and the repository's
check-leaks.shwith the private-strings list over the complete final findings comment and every complete final follow-up body, including embedded corpus and command output. Also obtain a separate leak review of the complete final text of each post before it is posted. Inspect the findings comment's recorded clean results for both scans and that post's leak review, covering every post; recheck any changed text. This verifies D4's three requirements.
Risks
A wrong row would let the Feature turn a failure green; AC3 and AC4 exist to catch that, and the Feature's Verify plan replays every row. The sweep might miss a class; AC1 names the known ones and the Feature adds a case for any a reviewer finds. The undo is deleting the throwaway branch; nothing else changes.
Operator authority
None
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Work type
Projects
- StatusShow more project fieldsBacklog
Activity
tbhb-agent commentedon Oct 25, 2026
Refinement: NOT READY — clarify observable behavior, integration verification, and rollout prerequisites
Checked
packages/agents/source/skills/merge-pr/shared/scripts/pull-request.sh:69,internal/checks/checks.go:180,internal/checks/checks.go:257).AC<n>:entries, within the limit, including a non-functional criterion. Implementation detail, compound outcomes, and verification gaps remain below.Nonesatisfies the required form. Clarify separately how the epic’s owner-approved protection prerequisite will be satisfied.Gaps
Acceptance criteria contain implementation prescriptions and multiple outcomes. AC4 mandates a Go command and deletion of a named function; AC5 mandates a specific command and generated implementation absence. AC1 combines contract publication with exhaustive difference accounting. Move implementation choices into Approach and separate each observable acceptance outcome.
The compatibility evidence is narrower than the migration risk. AC2 checks only formerly green rollups, and its Verify entry replays existing Bats fixtures. Require contract cases for all identified differences, with expected old and chosen outcomes, including formerly failing and pending inputs. Otherwise, a failure becoming success can escape that comparison. Existing Go tests already provide readable evidence for ties, repeated contexts, unknown states, and required-check behavior (
internal/checks/checks_test.go:63,internal/checks/checks_test.go:78,internal/checks/checks_test.go:94).Approach omits the caller output and exit-status transition. Shell emits per-check TSV with states and workflow-qualified names; Go emits summary JSON, with unreadable input returning 5 instead of 3 (
packages/agents/source/skills/merge-pr/shared/scripts/pull-request.sh:49,internal/agentrepo/command.go:45). Preflight formats those TSV rows and reports superseded runs; merge execution uses them for failure and pending diagnostics (packages/agents/source/skills/merge-pr/shared/scripts/preflight.sh:101,packages/agents/source/skills/merge-pr/shared/scripts/squash-merge.sh:203). State which outward diagnostics remain and which deliberately change. Verify both callers’ output and refusal behavior against the branch binary.AC3 lacks merge-path failure verification. Go table cases alone do not prove callers refuse unsafe inputs or handle an unavailable command. Add integration checks for empty, unreadable, truncated, and unknown-state rollups plus command failure, asserting no merge write occurs. Define whether “unknown-state rollup” includes an unknown state in a superseded entry: Go validates verdicts after selecting latest entries (
internal/checks/checks.go:148,internal/checks/checks.go:257).AC6’s benchmark does not establish its threshold by itself. Specify the benchmark invocation, measurement boundary, and explicit comparison proving a 500-check evaluation finishes under one second.
Dependencies omit rollout protections required by Replace complex shell with tested Go, starting with gates and merging #588. Identify how consumers receive the required binary before updated callers, and how owner-approved protection mapping covers changed Go enforcement paths before rollout. Current configured golden and owner patterns omit those Go paths (
.agents/agent-skills.conf:4,.agents/agent-skills.conf:9). If another issue supplies either prerequisite, name it and add the appropriate native blocker; otherwise distinguish work included here from a rollout hold.Generation and consumer verification are incomplete. AC5 covers generated-skill consistency, but the plan omits the full generation → token measurement → deployment sequence and consumer migration documentation. Add those checks and document adoption steps or explain “No consumer impact.” Replacing this judge also does not remove jq from the surrounding merge path (
packages/agents/source/skills/merge-pr/shared/scripts/pull-request.sh:19).Split
No split required. Keep the contract, evaluator adjustments, caller transition, focused regression coverage, generated output, and adoption guidance together. Reference independently delivered rollout prerequisites explicitly.
Definition of done
No item is fully covered by the current AC and Verify plan; implementation and merge evidence remain pending.
mise run check,mise run cover-go-check, andmise run cover-py-checkare absent.Round: 1 of 3
tbhb-agent commentedon Oct 25, 2026
Refinement: NOT READY — native prerequisite and caller compatibility gaps remain
Checked
packages/agents/source/skills/merge-pr/shared/scripts/pull-request.sh:69,internal/checks/checks.go:180,internal/checks/checks.go:257).Nonesatisfies the required form. The separate owner-approved protection hold is now explicit; current patterns still omit replacement Go paths (.agents/agent-skills.conf:4,.agents/agent-skills.conf:9).Round 1’s benchmark, generation sequence, adoption guidance, and identification of rollout protections are addressed. The remaining gaps follow.
Gaps
Record the stated merge prerequisite natively. Dependencies says this change “must not merge ahead of” Ship the Go binaries as release assets installed through mise's github backend #333, which remains open, but declares “Native blocked-by: none.” Add Ship the Go binaries as release assets installed through mise's github backend #333 as a native blocker. Keeping the protection update as an explicitly separate rollout hold is consistent with Replace complex shell with tested Go, starting with gates and merging #588.
Resolve superseded-run reporting before deleting its implementation. Approach deletes
mp_superseded, but AC5 covers only failing or running checks and their state/name output. Preflight also prints the number of omitted earlier runs, with an existing assertion requiring that message (packages/agents/source/skills/merge-pr/shared/scripts/preflight.sh:110,tests/skills/merge-pr/scripts/merge-workflow.bats:75). Require preservation through the replacement, or an explicit contract decision to change that diagnostic, with integration coverage.Verify the complete compatibility table and existing Go consumer. Approach makes the contract table a fixture, but AC2’s replay is limited to existing merge Bats rollups; AC3 adds selected cases. Require execution of every contract row against its recorded old and chosen outcomes, including malformed inputs, superseded unknown states, and head movement. Repeated status contexts already demonstrate a concrete unsafe migration: Go accepts an older ERROR followed by newer SUCCESS, while shell retains the ERROR (
internal/checks/checks_test.go:78,packages/agents/source/skills/merge-pr/shared/scripts/pull-request.sh:89). Also retain regression coverage for unattended evaluation:ma_gatesalready invokes the same Go command with required names and expected head (packages/agents/source/skills/merge-pr/shared/scripts/merge-agent.sh:169). Shared evaluator changes must preserve that consumer’s required-check and head-binding behavior.Cover command failure beyond absence, and describe preflight accurately. AC4 covers a judge that cannot run, but its Verify entry tests only a missing command. Add a failing command stub, including partial success-looking output followed by nonzero exit, and assert no merge write. AC5 also speaks of each caller’s “refusal” status, although preflight currently reports failed checks and exits successfully (
tests/skills/merge-pr/scripts/merge-workflow.bats:86); squash execution refuses with status 1 (packages/agents/source/skills/merge-pr/shared/scripts/squash-merge.sh:56). State and verify those distinct outward contracts.Finish separating observable AC from implementation. AC1 still combines contract publication with exhaustive difference accounting; AC6 combines implementation deletion with reproducible generation and names internal functions indirectly through its Verify entry. Move deletion prescriptions into Approach and give independently verifiable outcomes separate AC entries. There is room within the twelve-criterion limit.
Split
No split required. Keep the compatibility contract, evaluator changes, caller adapters, integration coverage, generated output, and adoption documentation together. Track #333 natively and retain the separate protection rollout hold.
Definition of done
Ticks indicate coverage in the proposed AC and Verify plan, not completed implementation evidence.
mise run check,mise run cover-go-check, andmise run cover-py-checkare absent.Round: 2 of 3
tbhb-agent commentedon Oct 25, 2026
Refinement: NOT READY — compatibility and failure contracts still leave unsafe or contradictory outcomes
Checked
Reviewed #720, its two previous verdicts, #588, native dependencies, #333, #544, and sibling scopes #713–#722. Code reads used only
origin/mainat56543d2. No files, checks, or external state were changed.AC<n>:entries, within the limit. AC7 supplies a non-functional criterion. AC1 and AC6 still mix acceptance outcomes with implementation and verification prescriptions.Nonesatisfies the required form.Round 2’s native #333 blocker and superseded-run diagnostic gaps are closed. Full-table execution and unattended regression coverage are now required. Failure-command coverage and separation of the two caller contracts improved, but remain incomplete.
Gaps
AC1 and AC6 still fail the requested acceptance-criterion structure. AC1 combines publication of a contract, its contents, and how tests execute it. AC6 combines deletion of an implementation with reproducible generation. Move test machinery and deletion prescriptions into Approach/Verify; express compatibility against the contract and reproducible generated output as separate observable outcomes. This was an explicit Round 2 finding and remains unresolved.
AC4’s unknown-state contract conflicts with the proposed exit mapping, and superseded unknown states remain unspecified. Go currently returns
Failed/1 for an unknown status, conclusion, or context state, rather thanUnreadable/5 (internal/checks/checks.go:180,internal/checks/checks_test.go:82,internal/checks/checks_test.go:120). Approach maps 1 to the ordinary failing-check path, but AC4 requires “could not be read.”Go also selects latest runs before judging states (
internal/checks/checks.go:148,internal/checks/checks.go:155). An older unknown-state CheckRun followed by a newer successful run therefore becomes green. AC4’s unqualified “rollup with an unknown state” requires refusal, but neither the required corpus classes nor Verify explicitly cover this superseded case. State whether validation covers every entry, define the attended diagnostic mapping, and test unknown status, conclusion, and context state—including superseded entries. Preserve or explicitly account for the existing unattended interface.The compatibility corpus still omits a concrete failure-to-green difference: timestamp ordering. Shell compares
startedAtstrings; Go parses timestamps and compares instants (packages/agents/source/skills/merge-pr/shared/scripts/pull-request.sh:71,packages/agents/source/skills/merge-pr/shared/scripts/pull-request.sh:88,internal/checks/checks.go:110,internal/checks/checks.go:269).For two completed runs of the same check, a failure at
2026-10-01T02:00:00+02:00and success at2026-10-01T01:00:00Zmake shell choose failure and Go choose success. That violates AC3, yet none of AC1’s required classes forces this case into the corpus. Invalid timestamps are another existing difference already demonstrated byinternal/checks/checks_test.go:158. Add timestamp-format and ordering cases, and specify a rule that cannot weaken the shell outcome. A corpus-only difference detector cannot discover classes absent from its corpus.“Preflight never refuses” is factually wrong and could erase existing safety behavior if implemented literally. Preflight calls
mp_require_reviewsbefore reporting checks and runs unattended gates before reaching the check section (packages/agents/source/skills/merge-pr/shared/scripts/preflight.sh:94). Required reviews cause exit 75; an existing regression explicitly requires this (tests/skills/merge-pr/scripts/merge-workflow.bats:385).Limit AC5 and Approach to the attended check-reporting section: failing or pending checks do not themselves make that section refuse. Preserve existing review, environment, and unattended-gate refusals. Verify both the reporting behavior and these existing refusal paths.
AC4’s failure messages cannot be established solely by inserting the proposed judge adapter. Both callers parse the saved response before invoking the judge. Preflight formats state and reads review status first; squash execution reads head and base fields first (
packages/agents/source/skills/merge-pr/shared/scripts/preflight.sh:78,packages/agents/source/skills/merge-pr/shared/scripts/squash-merge.sh:116,packages/agents/source/skills/merge-pr/shared/scripts/pull-request.sh:39). A truncated or malformed live response can exit there with a parser error before the promised “could not be read” message.Missing or unusable commands can also fail earlier in
ma_context, with an existing diagnostic that AC9 promises to preserve (packages/agents/source/skills/merge-pr/shared/scripts/merge-agent.sh:28,tests/skills/agent-merge-pr/workflow.bats:12). Clarify which entry points and input boundaries AC4 covers, reconcile its generic diagnostic with AC9, and ensure integration tests reach and assert the intended boundary rather than merely observing an earlier refusal.The subprocess verification still omits Round 2’s success-looking partial-output case. Verify AC4 now names an unmapped exit status and garbage output, but does not explicitly exercise a command that emits a valid-looking success result and then exits with a mapped nonzero status, such as 1 or 2. Garbage output does not establish that the adapter checks process status when parsing succeeds.
Add this disagreement case, require nonzero status to prevent success, and assert the diagnostic and absence of a merge call. Also reject a zero-status response lacking the required result fields. The command’s result and process status are distinct parts of its interface (
internal/agentrepo/command.go:68,internal/agentrepo/command.go:74,internal/agentrepo/command.go:96).Binary publication is now blocked correctly, but installation of a compatible binary is still not secured. Ship the Go binaries as release assets installed through mise's github backend #333 explicitly excludes consumer installation; feat(distribution): migrate consumer tool pins to release assets #544 owns that work. Existing adoption guidance permits no
agentrepo, and the shared loader accepts releases from v0.16.0 upward (docs/agent-skills-migration.md:9,packages/agents/source/lib/lib.sh:140). Those releases do not provide the proposed diagnostic listing.Add feat(distribution): migrate consumer tool pins to release assets #544 as a native prerequisite, or specify and verify an alternative installation sequence. In either case, identify the release/capability required by the new callers and ensure it is installed before deployment. Publishing the first archive alone does not establish that consumers have the replacement interface. AC8’s documentation inspection does not prove installation or compatibility.
Split
No mandatory split identified. Keep the contract, corpus, evaluator changes, caller adapters, generated output, and adoption documentation together. Keep binary distribution, consumer installation, and the owner-protection rollout update explicitly coordinated with their separate delivery work.
Definition of done
The definition of done for #720 (Feature). Tick each item and give its evidence.
check-leaks.shwith the repository's private-strings list, and the leak review of each post.docs/, CHANGELOG or help text), or the Verification section says "No doc impact" and why.provenance:agent-assistedlabel.Status:comment names the merge commit.mise run checkis green, and so are the coverage floorsmise run cover-go-checkandmise run cover-py-check, whichcheckdoesn't run.packages/agents/source/commits whatmise run generate-skills, thenmise run skill-tokens, thenapm installrewrite, in that order, andcheck-generated-skills(part oflint) passes.docs/agent-skills-migration.mdor the adoption docs, or the PR's Verification says "No consumer impact" and why.Round: 3 of 3
[-]Port the shell check judge to Go and settle one contract[/-][+]Settle the check judge contract and divergence corpus before porting it[/+]tbhb-agent commentedon Oct 25, 2026
Status: redesigned after round 3 of 3 (#720 (comment))
Done:
Why: round 3 found that one issue mixed a contract that was not yet settled with the migration that depends on it, and that the failure and compatibility rules were unsafe or contradictory in places. The Spike settles the contract first.
Next: refinement round 1 of 3 for this Spike and for #723.
Blocked on: nothing for this Spike. #723 is blocked on this Spike, #333 and #544.
tbhb-agent commentedon Oct 25, 2026
Refinement: NOT READY — bound the sweep and make release and refusal evidence achievable
Checked
Read #720, all four comments, #588, native links, #723, #333, #544, #713 and the additional native dependent #719. Code references are to
mainatd41369e7. No files, checks, repository tooling, posts or operator commands were run.packages/agents/source/skills/merge-pr/shared/scripts/pull-request.sh:69; Go validation, instant ordering and verdict selection appear atinternal/checks/checks.go:75,internal/checks/checks.go:110,internal/checks/checks.go:180andinternal/checks/checks.go:257. Caller reporting, early parsing and review refusal also match the description.AC<n>:entries, within the limit. Each describes a findings artifact or recorded experiment; AC10 covers performance. The implementation-deletion prescriptions from the former Feature are gone.Noneis appropriate.Gaps
AC3 leaves the sweep’s domain undefined, and Verify AC4 does not prove its universal claim. “Every combination” does not specify which known states, which state fields, which entry types, or which timestamp values constitute the finite sweep. Status contexts use
createdAt, while CheckRuns usestartedAt(internal/checks/checks.go:110); a generic “start-time spelling” axis leaves that distinction to the researcher.Require the findings to enumerate the domains, generation rule and expected total before reporting coverage. For AC4, two example rows cannot establish the outcome for any pair whose string and instant orders disagree. Require an explicit general decision rule, supported by the defined sweep and code reasoning. Also reconcile AC3’s accepted-exception option with Approach’s unconditional “never greener” rule.
AC7’s Verify entry does not cover all the refusals its table must inventory. It names the two entry scripts,
pull-request.shand the review regression, but early stops also come from the shared loader and unattended helpers. Examples include the loader’s version refusal atpackages/agents/source/lib/lib.sh:151, command usability atpackages/agents/source/skills/merge-pr/shared/scripts/merge-agent.sh:28, and unattended gates atpackages/agents/source/skills/merge-pr/shared/scripts/merge-agent.sh:118.Define the boundary as the shell check-reporting/judging section, distinguish the relevant invocation modes, and require tracing the helpers invoked before it. Otherwise “each refusal” can omit existing stops while passing the listed verification. The exit-75 review regression itself is correctly identified at
tests/skills/merge-pr/scripts/merge-workflow.bats:385.AC9 requires release evidence that may depend on the implementation this Spike explicitly excludes. The current evaluator can choose success for the offset example that AC4 requires to remain non-green (
internal/checks/checks.go:269). Naming and demonstrating a release that implements the chosen future contract therefore cannot be assumed achievable before Replace the shell check judge in the merge scripts with the Go evaluator #723.Allow an explicit “no compatible release exists yet” finding. In that case, require the capability contract, the issue responsible for implementing and releasing it, and the installation sequencing through feat(distribution): migrate consumer tool pins to release assets #544. Verify that recorded decision rather than requiring a published compatible release to close an otherwise unblocked Spike.
D4 has no verification step. The plan does not require gitleaks,
check-leaks.shwith the repository’s private-strings list, or leak review of the findings and follow-up posts. Add those checks and their recorded results to the completion plan. The recorded corpus and command output are part of the material being published.Split
Keep the Spike/Feature split. #720 should settle and record the contract and corpus; #723 should implement and verify the migration. Release production and installation remain coordinated through #333 and #544. No further split or pull request is required for this Spike.
Definition of done
Ticks indicate coverage by the AC and Verify plan, not completed execution.
PR items D1–D3 and D5–D10 do not apply.
Round: 1 of 3
tbhb-agent commentedon Oct 25, 2026
Refinement: NOT READY — clarify the sweep’s state fields, complete the caller contract, and finish D4 coverage
Checked
Read the current #720 body, all comments including the Spike’s round 1 verdict, #588, native links, #723, #333, #544 and #719. Code was read from
mainatd41369e7.packages/agents/source/skills/merge-pr/shared/scripts/pull-request.sh:69; Go validation, state handling and instant ordering appear atinternal/checks/checks.go:75,internal/checks/checks.go:180andinternal/checks/checks.go:269. Early parsing, caller reporting and exit-75 review refusal also match.AC<n>:entries. They describe findings artifacts or recorded experiments, with non-functional coverage in AC10 and AC12. The remaining contract and coverage gaps are below.tests/skills/merge-pr/scripts/merge-workflow.bats:385.Noneis appropriate.Gaps
AC3 still leaves the CheckRun state domain ambiguous. The two entry types and seven timestamp relations now bound much of the sweep, but a CheckRun has two independently relevant fields:
statusandconclusion. “Every state spelling for its type” does not specify whether their domains are crossed, or whether one field varies while the other stays fixed. The judges branch on status before reading conclusion (packages/agents/source/skills/merge-pr/shared/scripts/pull-request.sh:75;internal/checks/checks.go:184).Specify the per-field domains and their combination rule, including which lowercase and missing-field variants participate. Otherwise, different generators can report different expected totals while claiming full coverage. Verify AC3 should check that field-level product, rather than only agreement between the researcher’s chosen total and recorded total.
The tables do not require the complete chosen caller contract that Replace the shell check judge in the merge scripts with the Go evaluator #723 consumes. AC1 requires the shell’s exit status, but only the chosen merge-path outcome. AC6 similarly maps command responses to outcome categories without requiring each caller’s process status, report or diagnostic. AC5 supplies messages and statuses for unknown states; AC7 supplies them for earlier stops. That leaves cases such as empty or missing rollups and malformed evaluator output without a required chosen status and diagnostic for each caller.
These distinctions matter: preflight reports unreadable or empty checks at
packages/agents/source/skills/merge-pr/shared/scripts/preflight.sh:104, while squash execution refuses atpackages/agents/source/skills/merge-pr/shared/scripts/squash-merge.sh:215. Replace the shell check judge in the merge scripts with the Go evaluator #723 AC3 and AC4 explicitly depend on the contract naming caller behavior.Require the relevant table rows to record preflight and squash outcomes separately, including process status and output/message, and make Verify inspect those fields. This completes the findings artifact without prescribing implementation.
Round 1’s D4 gap is only partly closed. AC12 and Verify AC12 now require gitleaks and the repository leak check with the private-strings list. Neither requires the separate leak review of each post named in D4.
Add that review for the complete findings comment and every follow-up body, including embedded corpus and command output, and require its clean result to be recorded.
Split
Keep the existing split: #720 settles the contract and corpus; #723 implements the migration. Distribution and installation remain with #333 and #544. No further split or pull request is required for this Spike.
Definition of done
Ticks indicate coverage by the AC and Verify plan, not completed execution.
A Spike needs no pull request. PR items D1–D3 and D5–D10 do not apply.
Round: 2 of 3
tbhb-agent commentedon Oct 25, 2026
Refinement: READY — all three round 2 gaps are closed
Checked
Read the current #720 body, its round 2 verdict, #723, #333, #544 and #719. Code references below are from
origin/mainatbfd9f84.packages/agents/source/skills/merge-pr/shared/scripts/pull-request.sh:69and Go’s parsing, verdict and timestamp logic atinternal/checks/checks.go:75,internal/checks/checks.go:180andinternal/checks/checks.go:269.6 + 6 + 1 + 1 = 14, conclusion9 + 9 + 1 + 1 = 20, and context state5 + 5 + 1 + 1 = 12. Thus7 × ((14 × 20)² + 12²) = 548800 + 1008 = 549808, exactly the stated total. Omissions and excluded alternatives are explicit.packages/agents/source/skills/merge-pr/shared/scripts/preflight.sh:101andpackages/agents/source/skills/merge-pr/shared/scripts/squash-merge.sh:203. The exit-75 regression exists attests/skills/merge-pr/scripts/merge-workflow.bats:385.Noneis consistent with the research scope.Gaps
None. The field-domain ambiguity, incomplete caller contracts and missing per-post leak review are closed. No new readiness gap or contradiction found.
Split
Keep the existing split: #720 settles the contract and corpus; #723 implements migration. Distribution and installation remain with #333 and #544.
Definition of done
Ticks indicate AC and Verify-plan coverage; execution remains pending.
check-leaks.shwith the private-strings list, and a separate leak review of every complete final post, with clean results recorded and changed text rechecked.A Spike needs no pull request. PR items D1–D3 and D5–D10 do not apply.
Round: 3 of 3
tbhb-agent commentedon Oct 26, 2026
Findings for #720: conservative check evaluation contract
Research head:
f130498bd7383fe02f8b9d21965f555866abd98c.I fetched origin and used an isolated worktree. No production files
changed. No commit, PR, merge or release forms part of this Spike.
The recommended contract validates every entry, keeps every status
context, and takes the worse check-run verdict selected by either
text order or instant order. A caller accepts success only when both
the process status and a complete, consistent result say success.
Evidence and notation
All source locations below refer to the research head. Let
Mmeanpackages/agents/source/skills/merge-pr/shared/scripts/andLmeanpackages/agents/source/lib/. Paths written after those prefixesconcatenate with them. This avoids ambiguous generated-copy
references.
M/pull-request.sh:69-110: shell judge and superseded count.internal/checks/checks.go:75-177: parse and evaluation precedence.internal/checks/checks.go:180-206,257-286: verdict and selection.internal/agentrepo/command.go:55-101,107-119: command protocol.M/preflight.sh:40-120: preparation, review and check reporting.M/squash-merge.sh:63-218: preparation and check refusal.M/merge-agent.sh:19-30,118-180: unattended context and gates.L/lib.sh:140-183,679-714,830-885: loader, stamps and API errors.L/agent.sh:56-65: interrupted operations and detached HEAD.The evidence block at the end is a lossless recording, not a link to
local files. Its metadata holds every hand-written input, both judges'
exact stdout, stderr and numeric status, full caller recordings,
unattended recordings, benchmark samples and the sweep manifest.
The binary matrix records both exact outputs for every sweep key.
The decoder reconstructs all 549808 input/output records. No fixture
requires access to the research machine.
Pmeans attended preflight,Smeans squash with an explicitoperator confirmation.
Umeans unattended preflight.Gmeanssquash without explicit confirmation, including unattended squash.
These are fixture invocation modes, not confirmation grants.
For AC1 and AC6, earlier prerequisites and later gates pass, with all
GitHub writes stubbed.
Preaches its check-reporting section andcontinues to process exit 0.
Seither refuses there or reaches asuccessful stubbed merge and process exit 0. The check section alone
doesn't establish that a real merge is authorized. AC7 overrides this
boundary whenever a real earlier stop applies.
The following executable definitions specify complete caller streams.
mis the decoded evidence metadata. Concatenation adds no implicitwhitespace. Every newline is explicit. Empty stderr means
"".pfxincludes the checks heading.sfxends just before judging.ptailandstailinclude all otherwise successful later output.Those tails come from the recorded isolated fixture, not a proposed
change to production diagnostics.
Each tuple is
(caller process status, stdout, stderr).P(5)is an unreadable report and continuation, not green.S(5)refuses with no merge call.P(4)reports empty and continues.S(4)refuses.P(1,...)andP(2,...)report and continue.S(1,...)andS(2,...)print the relevant rows and refuse.The failure diagnostic deliberately drops “judged by its latest run”:
that phrase would misdescribe the conservative two-order decision.
AC1, AC2 and AC5: input contract and recorded comparison
Table keys are exact
idvalues inm['hand']. That record holds thesaved input bytes, command arguments, shell stdout/stderr/status and
current Go stdout/stderr/status. Both commands ran on each input.
All shell stderr streams were empty. Go stderr was empty except for
the required-check hints explicitly recorded under AC8.
The last column marks deliberate changes to the attended shell path.
yesincludes stricter interpretation or changed diagnostics.parseridentifies an earlier stop, not an adapter result.OK = [("SUCCESS", "ci / test")]. Input keys:malformed.missing.null.empty.success.failure.pending.unknown_status.unknown_conclusion.unknown_context.lower_status.lower_conclusion.lower_context.missing_status.empty_name.unknown_type.tie.tie_reverse.unstarted.repeated_context.offset.invalid_time.superseded_status.superseded_conclusion.superseded_context.For h00, both callers exit 5 before evaluating checks. The exact
stdout and stderr are in
m['callers']formalformed, with raw{as the response. Stderr is the parser diagnostic below. The preflight
first stops at its state formatter. Squash first stops in
mp_field.This result stays unchanged in the chosen contract.
Missing and null rollups deliberately change from empty to unreadable.
Their check-section reports are P(5) and S(5), not the old empty
reports. An actual empty array remains P(4) and S(4).
Unknown status, conclusion and context state all mean unreadable,
even on superseded entries. Validate every raw entry before selection,
including a conclusion that a non-completed status would otherwise
hide. Lowercase spellings and a missing CheckRun status are also
unreadable. This preserves Go's refusal and tightens shell acceptance.
Don't silently uppercase them and loosen the unattended path.
Accept the six uppercase CheckRun statuses and nine uppercase
conclusions in AC3. An absent, null or empty conclusion is allowed and
means pending when status is COMPLETED. An absent, null or empty
context state means PENDING. Recognized pending statuses still ignore
a recognized conclusion for verdict purposes. Null/empty CheckRun
status, other field types, empty names and unknown entry types are
unreadable. A context accepts only the five uppercase context states,
plus the missing-value convention above. These validation rules also
cover entries excluded by time ordering.
AC3: finite sweep, coverage and results
The domain manifest was written before execution. Generation order is
CheckRun then StatusContext, first-entry index, second-entry index,
then timestamp-pair index. CheckRun indices enumerate status as the
outer axis and conclusion as the inner axis. Omission is absence of
the JSON property. It isn't null or an empty string.
then those six lowercase spellings, UNKNOWN_STATE, then omission.
STALE, STARTUP_FAILURE, SUCCESS, TIMED_OUT, then those nine
lowercase
spellings, UNKNOWN_STATE, then omission.
those five lowercase spellings, UNKNOWN_STATE, then omission.
offset-reversed, invalid first, invalid second, both invalid.
m['manifest']['times']records every literal timestamp.head, CheckRun workflowci, nametest, or contexttest. No required names or expected-head argument. No mixed typesor extra fields. JSON array order is first then second.
Expected total, fixed before running:
7 * ((14 * 20)**2 + 12**2) = 549808.That is 548800 CheckRun pairs and 1008 context pairs.
The shell batch ran the exact jq program extracted from
mp_judge_checks. Its TSV rows went through the same failure-first,pending-second status reduction. The Go batch called the actual
agentrepo.Runcommand handler for each input, with fresh buffers andno constraints. Batching avoids over a million process launches.
The unmodified shell function and built CLI separately matched both
streams and statuses on 103 fresh-process samples, plus every hand
fixture. This distinguishes command recordings from an independent
reimplementation of either judge.
The coverage check independently reconstructed all mandated field
products and all expected keys, compared each entry's complete fields
with its index and checked timestamp values, order and fixed fields.
It found exactly 549808 unique keys, zero duplicates, zero missing
keys and zero unexpected keys. All 280 crossed CheckRun entries and
all 12 context entries participate in both positions.
Chosen results: 382 success, 2168 failing, 11994 pending and 535264
unreadable. Zero shell failures or pending results become success.
Current Go yields success for 120 shell-failing and 624 shell-pending
cases. The chosen rule prevents all 744. The full three-way counts
are in
m['summary']['cross'], whose rows are[shell status, current Go status, chosen status, count].AC4: time ordering rule
First reject every invalid timestamp, even on a superseded entry.
For each CheckRun group
(workflowName, name), compute both theshell text-order maximum and the parsed-instant maximum. Keep every
entry tied at either maximum. Evaluate those candidates and choose
failure over pending over success, with deterministic state/name
ordering for equal verdicts. The selected display row represents
that worst candidate. Count one displayed row per CheckRun group.
Use the shell's exact text key: missing/empty start or a start that
begins
0001maps to9999-12-31T23:59:59Z. For instant order,missing/empty start or the exact zero timestamp outranks started
runs. Other timestamps must parse as RFC3339Nano. This preserves the
shell's unusual zero-prefix case without letting it choose green
over a conflicting instant-order failure.
For any ordering disagreement, the candidate union includes both
maxima. Its worst verdict cannot be green if either maximum fails or
is unfinished. Ties retain all candidates, so array order cannot
weaken the result. The argument applies to groups larger than pairs.
For h20's offset example, the failure wins and gives P(1,F,1) and
S(1,F). The sweep found zero unsafe offset-reversed cases.
Keep every StatusContext entry, in its existing array order for
otherwise equal display names. Don't discard an older failing or
pending context. Validate
createdAt, but don't use it to discardcontexts. This preserves the shell contract for repeated contexts.
Retain the old superseded count: number of CheckRuns minus the number
of distinct
(workflowName, name)groups. Its legacy wording remainsexactly as in P above, including for ties. It counts collapsed display
rows, not proof that every omitted timestamp is strictly earlier.
AC6: evaluator process and result contract
Keep numeric outcomes 0 through 6 and their precedence: unreadable,
head moved, empty, missing required, failed, pending, success.
Require an additive
schema: 1,checksarray and nonnegativeinteger
supersededon every result. Each check has stringstateandnameand a
verdictofok,pendingorfail. Preserve existingoutcome, sorted uniquenames,skippedanderrorsemantics.For unreadable input require a nonempty
errorand empty checks.For missing required checks, keep the missing names in
names.Validate the whole stdout as exactly one JSON object with no trailing
non-whitespace bytes. Verify types, outcome/status agreement, listing
verdicts, outcome precedence and consistency of summary names and
counts. For success require a nonempty all-ok listing, no error or
non-success names, and process exit 0. This rejects empty success and
partial success-looking output. Don't infer success from an exit
status alone or from a JSON fragment.
Let
R(n)mean the complete, internally consistent schema-1 resultfor outcome n, ending with one newline. Specific result examples are
specified below in executable form. Unless stated otherwise evaluator
stderr is empty. Caller stdout/stderr/status use the full P/S tuples,
not the evaluator status. All these rows occur at the check section
unless the early-boundary column says otherwise.
garbage\n{}\n[]\n{"outcome":0}\nThe absent-command row describes P/S with the loader allowing absence.
U/G instead stop at
ma_context, as AC7 specifies. Loader versionrefusals precede both. Mapped 1 or 2 with R(0) remains unreadable,
not a valid failed/pending result. Evaluator stderr is captured and
must never substitute for a result. Keep recognized skipped-check
hints as described in AC8. For protocol/command failures, report the
stable unreadable message rather than copying arbitrary subprocess
stdout into the caller report.
For outcome 3, append MISSING_REQUIRED diagnostic rows for missing
names to the display listing. The P/S row above includes both the
reported successful check and
the missing-check diagnostic in preflight. Squash prints only the
missing-check diagnostic before refusing. Head-moved status 6 remains
a
separate evaluator outcome and refuses as unreadable in the attended
adapter. U preserves the earlier gate's status 6.
AC7: earlier stops, modes and exact stream rules
No earlier gate is removed or relaxed. Full process recordings for
malformed JSON and review/PR-state cases are in
m['callers']andm['early']. They include both streams, numeric status andreachability.
The remaining rows below are source-traced contracts. They aren't
claims that every external failure was injected into a live workflow.
For the source-traced rows, use the exact source prefixes below to
complete stdout. Messages in the diagnostic catalogue end in one
newline. An ordinary
dt_dieadds no prefix. Squash'sdieaddssquash-merge:. Both exit 1. Empty stdout/stderr is explicitly"".The table's tuple notation is
(status, stdout, stderr, reached).passmeans the other caller proceeds to P(0,OK) or S(0), with itsfull recorded stdout, empty stderr, status 0 and reached=yes.
With U enabled, append
report(number,status)to stdout on everynonzero exit after
pull-request.shinstalled its trap. Beforepreflight assigns its number,
numberisunknown. During loaderfailure, the trap isn't installed, so no report appears. G appends it
only when DEVTOOLS_UNATTENDED=1. A configured reporting transport
keeps its existing output and failure behavior instead. This is
orthogonal to the judge result and doesn't change the exit status.
The table uses ordinary P/S tuples and explicitly states mode changes.
All rows preserve their current stop and message. Source suffixes
refer to the catalogue immediately following the table.
PGis"\n== gh ==\ngh is not installed\n"followed by"Stop here and tell the operator.\n".PJreplacesghwithjq.PNis Pgh followed by the no-target block in preflight lines 66-74,with branch
topicand an empty open-PR list. Its exact bytes are:Diagnostic catalogue, using quoted Python strings for exact bytes.
binaryis the selected executable path,hereisharnessin therecorded fixture, and the example old version is v0.15.0. These are
parameters of the existing diagnostic, not text to hardcode in #723.
D1 includes the failed version command's exact stderr, empty when
that command printed none. D3 also has source-defined alternatives,
both P/U process exit 1
before the check section and S/G pass:
worktree setup: Vale styles are missing. Run:\n\n mise run devtools:worktree-setup\n, orworktree setup: could not check mise trust:\nplus captured outputand newline. Other Vale configuration failures use
worktree setup: Vale could not load the repository configuration:\nplus captured output and newline.
D4 also covers malformed expected-head syntax with
squash-merge: Expected head must be a full commit SHA\n.D13 is the exact
dt_require_stampdiagnostic followed by D13tail:changed draft, changed bound field, changed repository configuration,
or changed trusted policy authority. Each ends in newline. The
field-change template is
The {field} changed after the review (reviewed {old}, now {new}). Run the review script again.The otherliteral messages are in
L/lib.sh:687-709. Failures reading the stampor its digest keep the underlying stderr before D13tail and exit 1.
For external reader/parser errors, E means the exact captured stderr,
not a replacement message. Exit 2 above is the concrete external
failure fixture, and 128 is the Git failure fixture. Other subprocess
statuses propagate numerically unchanged. Parser status is jq's 5,
with its actual diagnostic and any partial stdout retained. An
arbitrary external program can emit arbitrary bytes, so this family
is parameterized by its captured bytes and numeric status. The h00
record gives a concrete complete parser example without that freedom.
RL is exactly the classifier message at
L/lib.sh:860-864, plusNo automatic retry was made.\n, followed by the original error.Secondary, primary GraphQL and primary REST have separate messages.
All exit 76 and retain original stderr. The first failing read decides
which stdout prefix applies. Later API reads use their prefix at that
point, not necessarily Pgh/S's empty prefix. No retry or budget query
belongs to this Spike.
U/G-specific earlier helpers follow. These modes execute ma_context.
P and explicitly confirmed S skip it and instead take their normal
report/merge paths, unless a common earlier gate above already stops.
For U, ma_gates runs before the check section. For G, ma_gates first
runs AFTER the check section and lint, at squash line 244. Don't move
its refusal earlier while replacing only the judge.
The --confirmed flag belongs to squash. The paired U invocation is
ordinary preflight, which reports and exits 0 with its normal stdout
and empty stderr when its gates pass. All other context stops have
reached=no for U/G. Plain G without the
unattended flag has empty stdout instead of report(...). Underlying
Git/config read failures propagate their exit status and stderr,
with the same empty prefix and trap rule. C6 includes Bash's
command -v uvoutput only if that builtin emits one in the selectedshell. Bash in this environment emits none for the lookup.
agent-merge-pr/shared/scripts/merge.shsets unattended mode andforwards to squash. With two numeric arguments it converts the second
into --expected-head. Its --confirmed refusal therefore comes from
the shared squash caller after ma_context, not from the wrapper.
Before U/G reach their initial response parsing,
mp_viewalso runsma_reviews. A malformed raw response stops at itsmp_fieldwithstatus 5, stdout Ptarget for U or Ss for G, and the exact jq stderr.
A failed review-pages read propagates its numeric status and stderr.
A mismatched review-page head stops with status 5 and the jq error
PR head changed while reading review authors; the check sectionisn't reached. P/S don't take this extra reader and continue normally
on a valid snapshot with the same paginated-review fault.
U's pre-check gates have the following first-stop catalogue. For
each row, U has reached=no. Its stdout is Pbefore plus any earlier
gate output plus the trap report. The stderr is the listed diagnostic
with newline. P and S pass on the same isolated gate-only fault.
G reaches the checks first, then stops at the same gate after lint:
its stdout is sfx plus the green line and the running-linters heading,
plus any prior gate output and its optional trap report. The process
status equals U's. This records the other caller even though its stop
lies beyond the check-section boundary.
A ma_gates review refusal is status 1. It isn't the status-75
mp_require_reviewsrefusal. That distinction stays unchanged.All ma_context checks run again inside ma_gates before this list.
A9 variants: unavailable repository URL, incomplete pinned histories,
failed complete diff, or failed name-status list. Preserve the exact
messages in
M/pull-request.sh:140-151and any preceding Git stderr.A10 is
merge policy unreadable: {reason}\n, from policy.py's mainexception handler, or a propagated uv/Python process failure. Reasons
cover invalid policy patterns, repository paths, APM lock fields,
review/findings shape, ruleset fields and missing required checks.
No raw policy value gets added to the replacement judge diagnostic.
Missing requirements from the policy reader use
merge policy unreadable: No required checks are known for the baseplus newline, rather than the later empty-output check's A12.
For owner-required policy, the existing JSON findings precede A11 on
stderr. A successful owner override instead prints its existing
owner override: {login} approved {head}\non stdout.Configuration parsing and Git/filesystem reads at any of these points
retain their numeric status and exact captured streams as above.
The malformed/required-review recordings and source trace establish
that preflight's CHECK SECTION reports failing and running checks
without refusing. They do not assert that preflight never refuses.
The existing review regression is at
tests/skills/merge-pr/scripts/merge-workflow.bats:385.AC8: unattended fixtures and hints
None of the existing check-bearing fixtures under
tests/skills/agent-merge-pr/changes its semantic outcome, numericstatus or stderr hint under this contract. The schema/listing fields
are additive stdout changes. The old summary fields keep their exact
values. The evidence includes direct evaluator replays of every
check-shape variant with required names and expected head supplied.
These are evaluator replays, not a claim that the entire Bats suite
ran through a future implementation.
Relevant fixtures:
workflow.bats:76,223,233,242,253,267,loosens.bats:28,confirmation.bats:33-34,161, and the base successfixture in
test_helper.bash. The missing-requirements policy casestops before evaluation with A12, unchanged. Missing/unusable/pinned
runner cases in workflow.bats:12-41 keep the AC7 boundaries.
In the compact rows,
test/testandtest/validatemeannames/skipped, each a single-name array.The full byte-exact old JSON and stderr are in
m['unattended'].Add the new fields without removing those summaries. Keep exact
required-name matching across every workflow. Required NEUTRAL and
SKIPPED fail, optional NEUTRAL and SKIPPED pass. Keep missing-required
precedence and the skipped hint. Head mismatch remains 6 and missing
expected-head data remains unreadable/5, as recorded in
m['hand'].Go unit fixtures
reordered statusesandstatus rerunininternal/checks/checks_test.godeliberately change success/0 tofailed/1. Unknown status/conclusion/context fixtures change failed/1
to unreadable/5. New coverage must add superseded invalid states,
lowercase fields, offset disagreements and command protocol faults.
These tightenings apply to both callers of the shared evaluator.
AC9: capability and release sequence
No compatible release exists yet. The current main implementation,
and the latest local tag v0.22.1, don't provide the proposed schema,
complete listing or superseded count. Main also still implements the
unsafe context/offset selection demonstrated above. Don't invent a
minimum released version for capabilities that don't exist.
Missing promised outputs: all-entry validation failures for unknown
and lowercase fields, conservative dual-order selection, all status
contexts, full workflow-qualified listing, superseded count, schema
marker and consistent protocol-failure adaptation. #723 implements
these evaluator and caller changes and documents the first release
that contains them. #333 supplies binary distribution. #544 installs
the compatible release before its dependent callers are deployed.
With no agentrepo, P reports unreadable/0 and S refuses unreadable/1
at the new adapter, while U/G retain their earlier C7 refusal/1.
A release below v0.16.0 still fails the shared loader with D2/1.
An older accepted release lacking schema 1 or its required fields
cannot be green: P reports unreadable/0, S refuses unreadable/1.
U/G must validate the same result protocol at their gate and refuse
with unreadable/5. An unusable executable can stop earlier at D1/C8.
Don't add a second shell judge as a fallback.
AC10: measured time and proposed limit
Generate 500 distinct successful CheckRuns, fixed head
head,workflow
ci, namestest-000throughtest-499, all started at2026-10-01T01:00:00Z. Save once. Build outside the timed region.Time each subprocess invocation through its completion, capturing
both streams, for 20 runs. All runs exited zero. The evidence stores
all 40 samples, in milliseconds.
Commands:
bash judge.sh 500.json(the unchanged sourced shellfunction) and
agentrepo checks evaluate --file 500.json.The timer used Python
time.perf_counter_ns(). These current-judgemeasurements exclude the not-yet-implemented schema validation and
caller adapter. #723 must measure that complete boundary separately.
Keep its proposed limit: median strictly below 1000 ms for 20 runs,
including invocation, whole-response validation and caller verdict,
with compilation excluded. Both existing medians are below the limit.
AC11: ownership, open questions and follow-up work
Every AC has a section in this comment. The migration belongs to
#723. It should
commit the contract, decoder and corpus before deleting the shell
judge, then implement the additive protocol and caller adapters.
Its tests should compare both caller streams and statuses, including
earlier-stop reachability. Update docs/checks-evaluator.md and the
consumer adoption documentation in that same feature.
Further work already has owners: #333 distribution, #544 installation,
#588 replacement-path protection, then #713 gate migration. #715,
#719, #721 and #722 retain their existing ordering and rollout holds.
No additional implementation issue is needed for findings already
inside #723. No duplicate issue was filed. No feature PR is
appropriate
for this Spike.
Open implementation choices that don't alter the contract: which
release number first carries it, how #723 stores the compact sweep
fixture, and whether its eventual diff requires the split already
specified in #723. The evaluator precedence, validation scope,
ordering rule, caller messages and failure mapping above are
decisions,
not questions deferred to the builder.
AC12: publication checks
PUBLICATION HOLD: the required private-strings list isn't available
in this session. No findings comment or follow-up body has been
posted. Gitleaks found no leaks in this complete draft and its decoded
metadata and 549808-record corpus. The repository check-leaks.sh also
returned zero, but warned that no private values were loaded. That
patterns-only result does not satisfy the required private-list scan.
A separate manual leak-review pass found no credentials, private
identifiers, local home paths or prohibited coordination references
in the complete draft or its decoded evidence. All fixture identifiers
are synthetic. No follow-up post exists. Recheck the final text after
adding the missing private-list scan result and before publication.
Recorded evidence and decoder
Save the following base64 block as
evidence.b64. Decode it with thecode below. The XZ payload starts with an unsigned little-endian
8-byte metadata length, then UTF-8 JSON metadata, then four bytes per
case: unsigned little-endian shell-dictionary index, one-byte
Go-dictionary index, one-byte proposed evaluator outcome.
Dictionary entries are
[status, stdout, stderr].Uncompressed SHA-256:
ee8e74ba494a5979c91eab8a563edf0c2ef758c2e1e0638fccd4891f70e5aa72.The matrix SHA-256 is in
m['summary']['matrix_sha256'].The chosen outcomes are a research model. They aren't represented as
implemented behavior. Replaying the recorded current outcomes uses
the cited head's shell function and built command. The Go package and
command tests passed with
go test ./internal/checks ./internal/agentrepo.The future implementation must pass the model and caller contract.
tbhb-agent commentedon Oct 31, 2026
Findings for #720: conservative check evaluation contract
Original recording head:
f130498bd7383fe02f8b9d21965f555866abd98c.Fresh replay head:
44be762a34eec4a832e1ed608c77ebf188357719.The corpus and original caller recordings extend the existing
research comment.
The judge corpus, hand fixtures and unattended evaluator fixtures were
independently replayed on the fresh head.
The recommended contract validates every entry, keeps every status
context, and takes the worse check-run verdict selected by either
text order or instant order. A caller accepts success only when both
the process status and a complete, consistent result say success.
Evidence and notation
Source locations in the inherited findings refer to the original recording head.
Caller supplements below use the fresh replay head. Let
Mmeanpackages/agents/source/skills/merge-pr/shared/scripts/andLmeanpackages/agents/source/lib/. Paths written after those prefixesconcatenate with them. This avoids ambiguous generated-copy
references.
M/pull-request.sh:69-110: shell judge and superseded count.internal/checks/checks.go:75-177: parse and evaluation precedence.internal/checks/checks.go:180-206,257-286: verdict and selection.internal/agentrepo/command.go:55-101,107-119: command protocol.M/preflight.sh:40-120: preparation, review and check reporting.M/squash-merge.sh:63-218: preparation and check refusal.M/merge-agent.sh:19-30,118-180: unattended context and gates.L/lib.sh:140-183,679-714,830-885: loader, stamps and API errors.L/agent.sh:56-65: interrupted operations and detached HEAD.The evidence block at the end is a lossless recording, not a link to
local files. Its metadata holds every hand-written input, both judges'
exact stdout, stderr and numeric status, full caller recordings,
unattended recordings, benchmark samples and the sweep manifest.
The binary matrix records both exact outputs for every sweep key.
The decoder reconstructs all 549808 input/output records. No fixture
requires access to the research machine.
Pmeans attended preflight,Smeans squash with an explicitoperator confirmation.
Umeans unattended preflight.Gmeanssquash without explicit confirmation, including unattended squash.
These are fixture invocation modes, not confirmation grants.
For AC1 and AC6, earlier prerequisites and later gates pass, with all
GitHub writes stubbed.
Preaches its check-reporting section andcontinues to process exit 0.
Seither refuses there or reaches asuccessful stubbed merge and process exit 0. The check section alone
doesn't establish that a real merge is authorized. AC7 overrides this
boundary whenever a real earlier stop applies.
The following executable definitions specify complete caller streams.
mis the decoded evidence metadata. Concatenation adds no implicitwhitespace. Every newline is explicit. Empty stderr means
"".pfxincludes the checks heading.sfxends just before judging.ptailandstailinclude all otherwise successful later output.Those tails come from the recorded isolated fixture, not a proposed
change to production diagnostics.
Each tuple is
(caller process status, stdout, stderr).P(5)is an unreadable report and continuation, not green.S(5)refuses with no merge call.P(4)reports empty and continues.S(4)refuses.P(1,...)andP(2,...)report and continue.S(1,...)andS(2,...)print the relevant rows and refuse.The failure diagnostic deliberately drops “judged by its latest run”:
that phrase would misdescribe the conservative two-order decision.
AC1, AC2 and AC5: input contract and recorded comparison
Table keys are exact
idvalues inm['hand']. That record holds thesaved input bytes, command arguments, shell stdout/stderr/status and
current Go stdout/stderr/status. Both commands ran on each input.
All shell stderr streams were empty. Go stderr was empty except for
the required-check hints explicitly recorded under AC8.
The last column marks deliberate changes to the attended shell path.
yesincludes stricter interpretation or changed diagnostics.parseridentifies an earlier stop, not an adapter result.OK = [("SUCCESS", "ci / test")]. Input keys:malformed.missing.null.empty.success.failure.pending.unknown_status.unknown_conclusion.unknown_context.lower_status.lower_conclusion.lower_context.missing_status.empty_name.unknown_type.tie.tie_reverse.unstarted.repeated_context.offset.invalid_time.superseded_status.superseded_conclusion.superseded_context.For h00, both callers exit 5 before evaluating checks. The exact
stdout and stderr are in
m['callers']formalformed, with raw{as the response. Stderr is the parser diagnostic below. The preflight
first stops at its state formatter. Squash first stops in
mp_field.This result stays unchanged in the chosen contract.
Missing and null rollups deliberately change from empty to unreadable.
Their check-section reports are P(5) and S(5), not the old empty
reports. An actual empty array remains P(4) and S(4).
Unknown status, conclusion and context state all mean unreadable,
even on superseded entries. Validate every raw entry before selection,
including a conclusion that a non-completed status would otherwise
hide. Lowercase spellings and a missing CheckRun status are also
unreadable. This preserves Go's refusal and tightens shell acceptance.
Don't silently uppercase them and loosen the unattended path.
Accept the six uppercase CheckRun statuses and nine uppercase
conclusions in AC3. An absent, null or empty conclusion is allowed and
means pending when status is COMPLETED. An absent, null or empty
context state means PENDING. Recognized pending statuses still ignore
a recognized conclusion for verdict purposes. Null/empty CheckRun
status, other field types, empty names and unknown entry types are
unreadable. A context accepts only the five uppercase context states,
plus the missing-value convention above. These validation rules also
cover entries excluded by time ordering.
AC3: finite sweep, coverage and results
The domain manifest was written before execution. Generation order is
CheckRun then StatusContext, first-entry index, second-entry index,
then timestamp-pair index. CheckRun indices enumerate status as the
outer axis and conclusion as the inner axis. Omission is absence of
the JSON property. It isn't null or an empty string.
then those six lowercase spellings, UNKNOWN_STATE, then omission.
STALE, STARTUP_FAILURE, SUCCESS, TIMED_OUT, then those nine
lowercase
spellings, UNKNOWN_STATE, then omission.
those five lowercase spellings, UNKNOWN_STATE, then omission.
offset-reversed, invalid first, invalid second, both invalid.
m['manifest']['times']records every literal timestamp.head, CheckRun workflowci, nametest, or contexttest. No required names or expected-head argument. No mixed typesor extra fields. JSON array order is first then second.
Expected total, fixed before running:
7 * ((14 * 20)**2 + 12**2) = 549808.That is 548800 CheckRun pairs and 1008 context pairs.
The shell batch ran the exact jq program extracted from
mp_judge_checks. Its TSV rows went through the same failure-first,pending-second status reduction. The Go batch called the actual
agentrepo.Runcommand handler for each input, with fresh buffers andno constraints. Batching avoids over a million process launches.
The unmodified shell function and built CLI separately matched both
streams and statuses on 103 fresh-process samples, plus every hand
fixture. This distinguishes command recordings from an independent
reimplementation of either judge.
The coverage check independently reconstructed all mandated field
products and all expected keys, compared each entry's complete fields
with its index and checked timestamp values, order and fixed fields.
It found exactly 549808 unique keys, zero duplicates, zero missing
keys and zero unexpected keys. All 280 crossed CheckRun entries and
all 12 context entries participate in both positions.
Chosen results: 382 success, 2168 failing, 11994 pending and 535264
unreadable. Zero shell failures or pending results become success.
Current Go yields success for 120 shell-failing and 624 shell-pending
cases. The chosen rule prevents all 744. The full three-way counts
are in
m['summary']['cross'], whose rows are[shell status, current Go status, chosen status, count].Fresh-head verification reconstructed all mandated keys, rejected duplicates,
and matched all 549808 shell triples, Go triples and chosen outcomes.
All 30 hand fixtures and 101 fresh-process sweep samples also matched.
Zero shell-failing or pending cases became green.
AC4: time ordering rule
First reject every invalid timestamp, even on a superseded entry.
For each CheckRun group
(workflowName, name), compute both theshell text-order maximum and the parsed-instant maximum. Keep every
entry tied at either maximum. Evaluate those candidates and choose
failure over pending over success, with deterministic state/name
ordering for equal verdicts. The selected display row represents
that worst candidate. Count one displayed row per CheckRun group.
Use the shell's exact text key: missing/empty start or a start that
begins
0001maps to9999-12-31T23:59:59Z. For instant order,missing/empty start or the exact zero timestamp outranks started
runs. Other timestamps must parse as RFC3339Nano. This preserves the
shell's unusual zero-prefix case without letting it choose green
over a conflicting instant-order failure.
For any ordering disagreement, the candidate union includes both
maxima. Its worst verdict cannot be green if either maximum fails or
is unfinished. Ties retain all candidates, so array order cannot
weaken the result. The argument applies to groups larger than pairs.
For h20's offset example, the failure wins and gives P(1,F,1) and
S(1,F). The sweep found zero unsafe offset-reversed cases.
Keep every StatusContext entry, in its existing array order for
otherwise equal display names. Don't discard an older failing or
pending context. Validate
createdAt, but don't use it to discardcontexts. This preserves the shell contract for repeated contexts.
Retain the old superseded count: number of CheckRuns minus the number
of distinct
(workflowName, name)groups. Retaining the "earlier/later"diagnostic for ties is intentional: its bytes stay exactly as in P.
The number counts collapsed display rows; it does not assert that
every omitted timestamp is strictly earlier.
AC6: evaluator process and result contract
Keep numeric outcomes 0 through 6 and their precedence: unreadable,
head moved, empty, missing required, failed, pending, success.
Require an additive
schema: 1,checksarray and nonnegativeinteger
supersededon every result. Each check has stringstateandnameand a
verdictofok,pendingorfail. Preserve existingoutcome, sorted uniquenames,skippedanderrorsemantics.For unreadable input require a nonempty
errorand empty checks.For missing required checks, keep the missing names in
names.Validate the whole stdout as exactly one JSON object with no trailing
non-whitespace bytes. Verify types, outcome/status agreement, listing
verdicts, outcome precedence and consistency of summary names and
counts. For success require a nonempty all-ok listing, no error or
non-success names, and process exit 0. This rejects empty success and
partial success-looking output. Don't infer success from an exit
status alone or from a JSON fragment.
Let
R(n)mean the complete, internally consistent schema-1 resultfor outcome n, ending with one newline. Specific result examples are
specified below in executable form. Unless stated otherwise evaluator
stderr is empty. Caller stdout/stderr/status use the full P/S tuples,
not the evaluator status. All these rows occur at the check section
unless the early-boundary column says otherwise.
garbage\n{}\n[]\n{"outcome":0}\nThe absent-command row describes P/S with the loader allowing absence.
U/G instead stop at
ma_context, as AC7 specifies. Loader versionrefusals precede both. Mapped 1 or 2 with R(0) remains unreadable,
not a valid failed/pending result. Evaluator stderr is captured and
must never substitute for a result. Keep recognized skipped-check
hints as described in AC8. For protocol/command failures, report the
stable unreadable message rather than copying arbitrary subprocess
stdout into the caller report.
For outcome 3, append MISSING_REQUIRED diagnostic rows for missing
names to the display listing. The P/S row above includes both the
reported successful check and
the missing-check diagnostic in preflight. Squash prints only the
missing-check diagnostic before refusing. Head-moved status 6 remains
a
separate evaluator outcome and refuses as unreadable in the attended
adapter. U preserves the earlier gate's status 6.
AC7: earlier stops, modes and exact stream rules
No earlier gate is removed or relaxed. Full process recordings for
malformed JSON and review/PR-state cases are in
m['callers']andm['early']. They include both streams, numeric status andreachability.
The catalogue is supplemented by fresh-head caller executions for stamp,
rate-limit, diff, policy and gate streams. Exact bytes are embedded below;
external commands use concrete failure fixtures.
For the source-traced rows, use the exact source prefixes below to
complete stdout. Messages in the diagnostic catalogue end in one
newline. An ordinary
dt_dieadds no prefix. Squash'sdieaddssquash-merge:. Both exit 1. Empty stdout/stderr is explicitly"".The table's tuple notation is
(status, stdout, stderr, reached).passmeans the other caller proceeds to P(0,OK) or S(0), with itsfull recorded stdout, empty stderr, status 0 and reached=yes.
With U enabled, append
report(number,status)to stdout on everynonzero exit after
pull-request.shinstalled its trap. Beforepreflight assigns its number,
numberisunknown. During loaderfailure, the trap isn't installed, so no report appears. G appends it
only when DEVTOOLS_UNATTENDED=1. A configured reporting transport
keeps its existing output and failure behavior instead. This is
orthogonal to the judge result and doesn't change the exit status.
The table uses ordinary P/S tuples and explicitly states mode changes.
All rows preserve their current stop and message. Source suffixes
refer to the catalogue immediately following the table.
PGis"\n== gh ==\ngh is not installed\n"followed by"Stop here and tell the operator.\n".PJreplacesghwithjq.PNis Pgh followed by the no-target block in preflight lines 66-74,with branch
topicand an empty open-PR list. Its exact bytes are:Diagnostic catalogue, using quoted Python strings for exact bytes.
The fixture executable is
agentrepo,hereisharness, and theexample old version is v0.15.0. These fixture values are not text to
hardcode in #723.
D1 includes the failed version command's exact stderr, empty when
that command printed none. D3 also has source-defined alternatives,
both P/U process exit 1
before the check section and S/G pass:
worktree setup: Vale styles are missing. Run:\n\n mise run devtools:worktree-setup\n, orworktree setup: could not check mise trust:\nplus captured outputand newline. Other Vale configuration failures use
worktree setup: Vale could not load the repository configuration:\nplus captured output and newline.
D4 also covers malformed expected-head syntax with
squash-merge: Expected head must be a full commit SHA\n.D13 is the exact
dt_require_stampdiagnostic followed by D13tail:changed draft, changed bound field, changed repository configuration,
or changed trusted policy authority. Each ends in newline. The
field-change template is
The {field} changed after the review (reviewed {old}, now {new}). Run the review script again.The literal prefixes and fullcaller streams are captured as
stamp-draft,stamp-field,stamp-configandstamp-policybelow. Unrelated trusted-ref movementwith unchanged policy content/status/authority no longer refuses.
Repository-config comparison applies only in unattended mode. Failures reading the stamp
or its digest keep the underlying stderr before D13tail and exit 1.
For external reader/parser errors, E means the exact captured stderr,
not a replacement message. Exit 2 above is the concrete external
failure fixture, and 128 is the Git failure fixture. Other subprocess
statuses propagate numerically unchanged. Parser status is jq's 5,
with its actual diagnostic and any partial stdout retained. An
arbitrary external program can emit arbitrary bytes, so this family
is parameterized by its captured bytes and numeric status. The h00
record gives a concrete complete parser example without that freedom.
RL is the complete stderr in
rate-secondary,rate-graphqlorrate-restbelow, including the original fixture error. All statusesare 76; P stdout is Pgh and S stdout is explicitly empty. For U/G,
append the trap report to stdout. Later failing reads use their prefix
at that read.
U/G-specific earlier helpers follow. These modes execute ma_context.
P and explicitly confirmed S skip it and instead take their normal
report/merge paths, unless a common earlier gate above already stops.
For U, ma_gates runs before the check section. For G, ma_gates first
runs AFTER the check section and lint, at squash line 244. Don't move
its refusal earlier while replacing only the judge.
The --confirmed flag belongs to squash. The paired U invocation is
ordinary preflight, which reports and exits 0 with its normal stdout
and empty stderr when its gates pass. All other context stops have
reached=no for U/G. Plain G without the
unattended flag has empty stdout instead of report(...). Underlying
Git/config read failures propagate their exit status and stderr,
with the same empty prefix and trap rule. C6 includes Bash's
command -v uvoutput only if that builtin emits one in the selectedshell. Bash in this environment emits none for the lookup.
agent-merge-pr/shared/scripts/merge.shsets unattended mode andforwards to squash. With two numeric arguments it converts the second
into --expected-head. Its --confirmed refusal therefore comes from
the shared squash caller after ma_context, not from the wrapper.
Before U/G reach their initial response parsing,
mp_viewalso runsma_reviews. A malformed raw response stops at itsmp_fieldwithstatus 5, stdout Ptarget for U or Ss for G, and the exact jq stderr.
A failed review-pages read propagates its numeric status and stderr.
A mismatched review-page head stops with status 5 and the jq error
PR head changed while reading review authors; the check sectionisn't reached. P/S don't take this extra reader and continue normally
on a valid snapshot with the same paginated-review fault.
U's pre-check gates have the following first-stop catalogue.
For these isolated faults, P and explicitly confirmed S continue with
P(0,OK) and S(0): status 0, their full defined stdout and explicitly
empty stderr. U stops before checks; G stops after checks and lint.
The supplement's
gatefunction gives exact full U/G tuples.Ordinary failures have no earlier gate stdout. HISTORICAL readiness
failures add exactly
{"outcome":0}\n; owner overrides add theirrecorded line before that JSON. CHOSEN FUTURE readiness failures add
R(0)from AC6 instead. Captured rows below specify historical streams.A ma_gates review refusal is status 1. It isn't the status-75
mp_require_reviewsrefusal. That distinction stays unchanged.All ma_context checks run again inside ma_gates before this list.
A9's five variants are
diff-url,diff-history,diff-fetch,diff-diffanddiff-namesbelow, including the fixture Git stderr.A10's concrete validation cases cover empty/invalid patterns,
repository paths, APM lock scalars, reviews, findings and applied rules.
policy-requireddistinguishes the reader's missing-requirementsmessage from A12's later empty-output refusal.
policy-processrecords process exit 2 and E.
owner-denyincludes the JSON findingbefore A11;
owner-overrideincludes the finding on stderr andoverride on stdout. No raw policy value enters the judge diagnostic.
On the fresh head ma_gates uses
agentrepo merge approval.No qualifying approval prints nothing before A6/A11.
approval-invalidrecords the missing-reviews command diagnosticbefore A6; the same command error precedes A11 for the owner check.
Caller status remains 1, U before checks and G after checks/lint.
The malformed/required-review recordings and source trace establish
that preflight's CHECK SECTION reports failing and running checks
without refusing. They do not assert that preflight never refuses.
The existing review regression is at
tests/skills/merge-pr/scripts/merge-workflow.bats:385.Exact caller supplements
These recordings execute fresh-head preflight and squash caller bodies with
their real stamp, rate classifier, diff and gate helpers. Prerequisite
readers pass; external failures and merge writes are stubbed, and filesystem
inputs use memory descriptors. Passing squash stops at the write boundary.
The real Python policy validator and exception handler supply validation
diagnostics; the lock-scalar case isolates that reader.
fixture failure\nis the injected process stderr, with explicitly emptyprocess stdout and exit 2. The snapshot has title
test, authorauthor,URL
https://example.invalid/pr/7, empty commits/files/body,one successful
ci / testCheckRun, and the values shown in stdout.Decode this zlib/base64 block as
ac7_b64with the Python below. Rows map each mode to[process_status, stdout_index, stderr_index], indexingstreams.These are HISTORICAL current-shell recordings, with the descriptor
label normalized to
fd 3;""is explicitly empty.Every nonzero row refuses. U rows have reached=no; gate G rows have
reached=yes. Stamp/rate/review-page rows have reached=no.
For stamp faults P/U pass; S uses G's tuple without its trap report,
except
stamp-config, which passes in S. For gate/policy/diff faultsP/S pass as defined above. Plain G removes the trap report only.
CHOSEN FUTURE stops and mode differences stay the same; evaluator
stdout gains the AC6 schema-1 fields. The reconstruction below defines
the future streams separately from the historical recordings.
A1–A8 and A12 use their identically named captured rows.
A9 uses the five
diff-*rows; A10 usespolicy-*;A11 uses
owner-deny; A13 usesready-executable.Readiness exit 2 uses
ready-status.owner-overridefollowsan approved owner finding with readiness exit 2 to record both streams.
CHOSEN FUTURE evaluator non-success uses
gate(mode,n,hint,R(n))for n=1..6, with AC8's preserved stderr hint;every byte of the complete AC6 result precedes the trap report.
Success evaluation before readiness prints exactly
eval_ok = R(0).Historical evaluator summaries, including AC8's, are not future results.
For readiness recordings, reconstruct the chosen future tuples as:
All other captured gate/policy/diff streams, statuses and stderr stay
unchanged.
gatealso reconstructs historical evaluator streams whenpassed their historical JSON explicitly.
For
reviews-head, jq's descriptor label is normalized tofd 3in both prose and decoded stderr; status is 5 for both callers and neither reaches
checks. P/S continue on the same isolated paginated-review fault.
Other external errors retain their captured numeric status and bytes.
AC8: unattended fixtures and hints
None of the existing check-bearing fixtures under
tests/skills/agent-merge-pr/changes its semantic outcome, numericstatus or stderr hint under this contract. The schema/listing fields
are additive stdout changes. The old summary fields keep their exact
values. The evidence includes direct evaluator replays of every
check-shape variant with required names and expected head supplied.
These are evaluator replays, not a claim that the entire Bats suite
ran through a future implementation.
Relevant fixtures:
workflow.bats:76,223,233,242,253,267,loosens.bats:28,confirmation.bats:33-34,161, and the base successfixture in
test_helper.bash. The missing-requirements policy casestops before evaluation with A12, unchanged. Missing/unusable/pinned
runner cases in workflow.bats:12-41 keep the AC7 boundaries.
In the compact rows,
test/testandtest/validatemeannames/skipped, each a single-name array.The full byte-exact old JSON and stderr are in
m['unattended'].Add the new fields without removing those summaries. Keep exact
required-name matching across every workflow. Required NEUTRAL and
SKIPPED fail, optional NEUTRAL and SKIPPED pass. Keep missing-required
precedence and the skipped hint. Head mismatch remains 6 and missing
expected-head data remains unreadable/5, as recorded in
m['hand'].Go unit fixtures
reordered statusesandstatus rerunininternal/checks/checks_test.godeliberately change success/0 tofailed/1. Unknown status/conclusion/context fixtures change failed/1
to unreadable/5. New coverage must add superseded invalid states,
lowercase fields, offset disagreements and command protocol faults.
These tightenings apply to both callers of the shared evaluator.
AC9: capability and release sequence
No compatible release exists yet. The fresh main implementation,
and the latest local tag v0.23.0, don't provide the proposed schema,
complete listing or superseded count. Main also still implements the
unsafe context/offset selection demonstrated above. Don't invent a
minimum released version for capabilities that don't exist.
Missing promised outputs: all-entry validation failures for unknown
and lowercase fields, conservative dual-order selection, all status
contexts, full workflow-qualified listing, superseded count, schema
marker and consistent protocol-failure adaptation. #723 implements
these evaluator and caller changes and documents the first release
that contains them. #333 supplies binary distribution. #544 installs
the compatible release before its dependent callers are deployed.
With no agentrepo, P reports unreadable/0 and S refuses unreadable/1
at the new adapter, while U/G retain their earlier C7 refusal/1.
A release below v0.16.0 still fails the shared loader with D2/1.
An older accepted release lacking schema 1 or its required fields
cannot be green: P reports unreadable/0, S refuses unreadable/1.
U/G must validate the same result protocol at their gate and refuse
with unreadable/5. An unusable executable can stop earlier at D1/C8.
Don't add a second shell judge as a fallback.
AC10: measured time and proposed limit
Generate 500 distinct successful CheckRuns, fixed head
head,workflow
ci, namestest-000throughtest-499, all started at2026-10-01T01:00:00Z. Save once. Build outside the timed region.Time each subprocess invocation through its completion, capturing
both streams, for 20 runs. All runs exited zero. The evidence stores
all 40 samples, in milliseconds.
Commands:
bash judge.sh 500.json(the unchanged sourced shellfunction) and
agentrepo checks evaluate --file 500.json.The timer used Python
time.perf_counter_ns(). These current-judgemeasurements exclude the not-yet-implemented schema validation and
caller adapter. #723 must measure that complete boundary separately.
Keep its proposed limit: median strictly below 1000 ms for 20 runs,
including invocation, whole-response validation and caller verdict,
with compilation excluded. Both existing medians are below the limit.
AC11: acceptance answers and follow-up
The caller migration and protocol implementation belong to
#723,
distribution to #333,
installation to #544,
and replacement-path protection to
#588.
No additional follow-up issue is needed.
Recorded evidence and decoder
Save the following base64 block as
evidence.b64. Decode it with thecode below. The XZ payload starts with an unsigned little-endian
8-byte metadata length, then UTF-8 JSON metadata, then four bytes per
case: unsigned little-endian shell-dictionary index, one-byte
Go-dictionary index, one-byte proposed evaluator outcome.
Dictionary entries are
[status, stdout, stderr].Uncompressed SHA-256:
ee8e74ba494a5979c91eab8a563edf0c2ef758c2e1e0638fccd4891f70e5aa72.The matrix SHA-256 is in
m['summary']['matrix_sha256'].The chosen outcomes are a research model. They aren't represented as
implemented behavior. Replaying the recorded current outcomes uses
the cited head's shell function and built command. The Go package and
command tests passed with
go test ./internal/checks ./internal/agentrepo.The future implementation must pass the model and caller contract.
AC12
Scans and leak review on the text above, before this section was added: gitleaks 8.30.1 (detect --no-git), exit 0, 0 hits; the devtools leak checker with the private-strings lists loaded, exit 0, 0 hits. Independent reviews: three rounds; the third found no blockers and a clean full-text leak review. The text above sha256 4875c68c0af60640b8f0fd23498acfebaa4462ec2ef8643af0f26d16c52a144e.