chore(agents): measured review and test strategy — mutation by tool, a self-filling ledger, fewer notes and tests

[V3RON]

Reviews and tests cost too much: ledgers nobody fills, notes nobody acts on, and specs whose Tests lines turn into 2–13 tests each. Each change was checked on real PRs.

• Faster weekly mutation. pnpm mutate: fix(gateway): a lone refusal after an aged-out window omits count (#677) #682 122→16 s, fix(cli): wait 20 s for a daemon a command auto-launched (#663) #674 265→21 s. Kept out of delivery: too slow for what it finds.
• Build one test at a time. One failing test: commit per spec line, then its code; observed for 3 deliveries.
• Ledger from transcripts. Prompts start with a ledger: tag; review-metrics.mjs reads tokens and cost from Claude Code's transcripts. It reproduces fix(cli): the auto-launch startup timeout reports DAEMON_STARTUP_FAILED, not INTERNAL (#672) #683's costs to the cent.
• Fewer notes, shorter reports. About 9% of 202 notes were acted on; notes remain only for an unproven secret leak or data loss.
• Fewer tests. One test per spec line, one level per claim, no new test for as-today behaviour, reviews ask only for required tests.
• Sharper audit probes. Would have caught 5 of 20 weak tests review found later.
• Protocol bump check. On push and in CI; flags all 7 past misses.
• recurring-findings routine. Classes fixed in 3+ PRs become draft tasks for a check.

Not included: the risk gate, which would have missed 21 fixed findings in 10 PRs.

Review

Spec review: skipped. Code review: 0 blocking, 0 fixed, 0 notes. Claims review: 0 blocking, 0 fixed, 0 notes.
Rounds: 0. Tokens: n/a.
Assessment: incomplete — not reviewed yet.

Written by an agent.

[V3RON]

Agentic code review: what others do, how we compare, and what PR #739 ships

2026-10-19. Goal: fewer tokens and less time per review, same or better quality.

Status after verification (read first). Parts of this report did not
survive testing on past PRs and are not in #739:

• Mutation in delivery (change 2): dropped. pnpm mutate caught 2 of 8
  past weak tests, 0 of 3 blocking ones. Call-delete and constant probes
  caught more, at about 15 minutes a file. Hand probes stay; the faster
  pnpm mutate runs only in the weekly harden-tests routine.
• Risk gate (change 4): dropped. It would have missed 21 fixed findings
  in 10 PRs.
• Testing: build mode is Pocock's alone. A second read of the sources
  (see "Testing: what each source says" below) found that Anthropic's own
  TDD workflow writes all tests first and freezes them, and that Osmani
  keeps a refactor step. Neither matches the build mode.

TL;DR

1. Replace hand mutation with the tool we already have. pnpm mutate <base>
   (scripts/mutate.mjs) already mutates only the lines a branch changed, with
   Stryker, and caches results per branch. That is what Google runs at review
   time. Today it is banned from delivery (toolchain.md:44). Hand mutation
   happens in up to four places (implementer audit, the Opus tests lens, the
   audit-tests appliers, harden-tests helpers), and the audit and the tests lens
   probe the same new tests twice. Run the tool once per PR. Its survivor list
   becomes the input to the fix loop and to the tests lens.
2. Find, then verify, in a cheap way. Every vendor uses the same shape:
   wide finders, then one verifier per finding that tries to disprove it, then
   only high-severity results get posted. We already have this. What we lack is
   a hard filter rule: "if not certain, do not report", and no notes.
3. Make depth depend on risk, and skip lenses more often. Anthropic's own
   numbers: PRs under 50 lines get on average 0.5 findings. Our recent small
   PRs (fix(cli): wait 20 s for a daemon a command auto-launched (#663) #674, fix(console): the overview's Activity lists the latest events, not the highest seq #680, fix(gateway): a lone refusal after an aged-out window omits count (#677) #682, fix(cli): the auto-launch startup timeout reports DAEMON_STARTUP_FAILED, not INTERNAL (#672) #683) ran three lenses and found nothing (fix(cli): the auto-launch startup timeout reports DAEMON_STARTUP_FAILED, not INTERNAL (#672) #683: six
   invocations for one routine change). A risk gate that runs one reviewer, or
   none beyond CI, on routine small diffs saves the most tokens.
4. Remove duplicate checks. Five checks run twice or more (list below).
   Each check gets one owner.
5. Measure, then tune. The ledger exists but is almost always empty
   (Tokens: n/a, cost unknown). Without numbers we cannot show any change is
   cheaper. The only recent full figure: feat(history): history.db, usage projections, simlock stats, GET /v1/stats and console figures (#623) #626 used 29M tokens on review.

What we will ship

Six changes, in this order. The ledger goes first, so every later change can
be measured against a baseline.

1. The ledger fills itself. Every agent prompt the deliver, review and
   implement skills write starts with a tag line:
   ledger: pr=<N> round=<1|2|fix-check> role=<role> [lens=<lens>].
   review-metrics.mjs --pr <N> finds the transcripts under
   ~/.claude/projects/ whose first message carries that tag. It works for
   subagents and for detached claude -p runs. For each one it records:

   • model and tokens, counting each message id once (one reply is written as
     several lines);
   • start and finish time;
   • role and lens, from the tag;
   • cost, from a price table we check in, with a date.

   merge-pr.sh runs the script and pastes its output into ## Review. No
   agent writes numbers by hand. A run the script can't see (opencode, cloud,
   transcripts deleted) shows as unknown, never zero.

2. Mutation by tool, once per PR. pnpm mutate <base> is allowed in
   delivery. The implementer audit runs it once after green, kills the
   surviving mutants or marks each equivalent with one line of reason, and
   writes mutants.txt into the review inputs. The hand probe of every test
   goes away. The tests lens stops probing; it checks the survivor decisions
   and the shape of the tests. Before the briefs change, we time it on 2–3
   past PR branches and set the line cap and timeout from that.

   Scope, as Google does at review time:

   • Only changed lines are mutated. mutate.mjs passes Stryker
     --mutate file:start-end ranges taken from the diff against the merge
     base.
   • Only covering tests run per mutant. Stryker has
     coverageAnalysis: "perTest".
   • Unchanged mutants are reused. --incremental keeps results across
     fix runs.
   • Still to check:

     • Done, 2026-10-19 (scripts/mutate.mjs, uncommitted). Stryker's
       first run took about 95% of the time, because Vitest's related mode
       follows imports to any depth. The script now runs in two passes:

       • A near pass runs only the test files that import a mutated file
         directly, through their own module's barrel, or that the branch
         changed.
       • A wide pass runs only for mutants the near pass left alive, with
         the full related set. The verdict matches a single related run.

       Measured, related run → two passes, same verdict each time: fix(gateway): a lone refusal after an aged-out window omits count (#677) #682
       122 s → 16 s, fix(leasing): no device work for a waiter the queue has just rejected (#665) #669 273 s → 16 s, fix(cli): wait 20 s for a daemon a command auto-launched (#663) #674 265 s → 21 s. A planted
       survivor took 235 s, because the wide pass ran 4,449 tests.

     • Cut noise as Google did: skip lines like logging and messages, and turn
       off mutators that keep producing useless survivors.

     • A tests-only PR changes no source lines, so there is nothing to mutate.
       It keeps a small hand-probe budget in the tests lens.

3. One owner per check.

   Check	Owner	Removed from
   Test title vs body (rule 10)	implementer audit	tests lens
   Stale-claims sweep	claims review	audit "Removed" step
   Done-when and scope	spec review	audit "Promised" step
   Rule breaches	spec review	tests lens Q2
   New tests fail when code breaks	pnpm mutate (change 2)	audit and tests lens probes

   Also fix the docs-only mismatch between REVIEW.md and calibrate-reviews.

4. A risk gate picks the lenses. A PR that is routine, under about 150
   changed source lines and touches no ownership, persistence, concurrency or
   trust-boundary code gets one Sonnet reviewer that covers spec and claims,
   plus CI and mutation. There is no Opus code lens on these. Every other PR
   is reviewed as today. The gate uses diff-kind.mjs and the risk reason we
   already record.

5. A higher bar and shorter reports. Reviewers report only what they are
   sure is real and matters, as P0 or P1. Each finding gives an exact
   file:line and a reproducer or a quoted rule. Lenses no longer write
   notes. Reports are capped at about 400 words, plus evidence as
   path:line refs.

6. Repeated findings become lint.

   • Record: the coordinator keeps the [class] each lens already gives
     (a rule such as testing 2, or at most 8 words). It records the class, outcome, file:line, reviewed commit and fix commit in
     the PR's ledger JSON, which is posted as the "Review ledger" PR comment.
   • Weekly pass: calibrate-reviews reads the ledgers of PRs merged in
     the last 4 weeks.
     • It counts confirmed (fixed) findings by class. A class fixed in 3 or
       more PRs is a candidate.
     • If a script can check the class, the pass opens a task for a lint rule
       (comment-lint, stale-refs or a new checker). The pre-fix and fixed code
       from those PRs become the rule's tests: the rule must flag the first
       and pass the second.
     • If a script can't check it, the pass adds one line to the brief or
       agent rule that owns it.
     • Rejected findings keep going to calibration.md, as today.
   • Data: today the pass would scrape free-text "Rejected:" lists from
     PR bodies, which only cover false positives. The ledger also holds the
     confirmed findings, and those are what become lint.

How we judge it: compare 10 PRs before and 10 after on tokens, cost,
wall time, findings fixed, findings rejected and bugs found after merge. If
bugs found after merge go up, change 4 is rolled back first.

Not shipping: more parallel finders or voting, and replacing Opus on
risky PRs.

Verification on real data (2026-10-19)

Every change above was checked against real PRs and transcripts before it
goes into a spec.

Change	Status	Evidence
1. Ledger	Verified	Prototype read #683's transcripts. All 5 rows that have a transcript match to within rounding ($0.051/0.0514, 0.053/0.0532, 0.193/0.1932, 0.263/0.263, 0.259/0.2595). The hand ledger also missed 4 of the 9 review sessions (≈$1.57 of ≈$2.39), and one of its rows (spec-r1b) has no transcript. A folder named review-683-claims held a spec run, so the role must come from the tag, not the path. Sonnet 5.5 cache reads cost $0.10/MTok, not $0.20; the skill's cached price table was stale. The token figure in a subagent's finish notice is its last context size, not its usage: one agent's notice said 93,773 when it used about 929k tokens.
2. Mutation by tool	Verified for speed and accuracy. Recall still being checked.	#682 122→16 s, #669 273→16 s, #674 265→21 s, all with the same verdict as one related run. Real bugs found while testing (--testFiles repeats, 0-based columns, a test-file pattern that missed 26 files) are fixed and covered by src/mutate.test.ts, 14 tests, each probed red. Still pending: would it have caught the weak tests that reviewers found by hand?
3. One owner per check	Partly confirmed	Replay of 116 PRs (one week): 13 explicit duplicate findings in 9 PRs. They are mostly spec + code, on "the test doesn't prove X" (8) and missing bounds (4). No recorded overlap on test titles or Done-when, so those two moves save effort but fix no observed duplicates.
4. Risk gate	Fails as proposed	At ≤150 changed lines, 27 of 81 PRs would be light (49 lens runs saved), but 21 fixed code-lens findings in 10 PRs would have been missed (#681, #667, #603, #586, #538, #645, …). At ≤50: 14 findings in 6 PRs. Most are "the test can't fail" or "untested branch" bugs. Line counts must also include vendor/ and scripts/: #645 has 96 src/ lines but 2,965 native lines, where 3 of its 4 bugs were. Revisit only after change 2 proves it catches this class.
5. Higher bar	Mostly confirmed	202 notes, about 18 acted on (≈9%). One was a real credential/endpoint leak (#625), confirmed only from notes. So drop routine notes but keep a path for security-type observations.
6. Lint from findings	Confirmed. Data source must change.	Classes in 3 or more PRs: stale comment or doc claim (20+), test can't fail (14), title vs body (9), protocol version bump (8), comment-rule breach (6), missing bound (5). Scriptable: protocol bump, comment rules, mutation survivors. PR bodies are reliable for counts only: 25 of 69 PRs with fixes list no individual finding. So the per-finding class must live in the ledger (change 1).

What the sources say

Anthropic

• Code Review (managed, March 2026): parallel finder agents → a
  verification step that removes false positives → dedup and rank by severity →
  post. It reads code only and never runs it. Under 1% of findings were marked
  incorrect. About 20 min and $15–25 per review. Depth grows with PR size:
  PRs over 1000 lines get 7.5 findings on average, PRs under 50 lines get 0.5.
  Cost advice: review once per PR, not on every push.
• REVIEW.md advice: at most five nits; skip anything CI enforces; per-path
  rule "only report if near-certain and severe"; every behaviour claim needs a
  file:line citation, not a guess from a name.
• Open-source /code-review plugin: models split by job. Haiku decides
  whether to review and finds rule files. Sonnet checks rule compliance. Two
  Opus agents hunt bugs, one with the diff only and one with the changed code.
  Then one validator per issue. It skips trivial, draft and already-reviewed
  PRs. Rule: "If you are not certain an issue is real, do not flag it." It never
  flags pre-existing issues, nits, lint, test coverage or input-dependent
  issues.
• Best practices: a reviewer with a fresh context. "A reviewer prompted to
  find gaps will usually report some, even when the work is sound" — so ask
  only for gaps that break correctness or the stated requirements.
• Multi-agent research post: multi-agent systems use about 15× the tokens
  of a chat. Coding splits into parallel parts less well than research.
  Subagents should return short summaries (1–2k tokens).

OpenAI

• "Scaling code verification" (Dec 2025): precision first, then recall.
  Post a comment only when its value is greater than the cost of checking it
  plus the cost of a false alarm. Repo access and running code beat a
  diff-only review. Verifying costs "a small fraction" of the tokens used to
  write the code. More budget mostly improved calibration (fewer false alarms).
  Authors changed code after 52.7% of comments.
• Codex on GitHub: flags only P0 and P1. Review rules are 2–3 short lines
  in AGENTS.md; formatting and lint stay in CI.
• Codex SDK cookbook: findings have a confidence score from 0 to 1, a
  priority from 0 to 3 and exact lines, plus one overall verdict on the patch.
  A new push cancels the review still running for the old one.
• Harness engineering (Feb 2026): almost all review is agent to agent,
  with few blocking gates: "corrections are cheap, and waiting is expensive".
  Repeated findings become custom linters whose error messages tell the agent
  how to fix them. Small cleanup PRs are reviewed in under a minute and
  merged automatically.

Addy Osmani

• Agentic Code Review (June 2026): risk sets the depth, not the author. "A
  config change earns a linter and a glance." CI is "the wall that does not
  move". Check edited tests before the code. "Coverage tells you a line ran,
  mutation testing tells you whether the test would notice." Different tools
  find different bugs: 93.4% of bugs were found by exactly one of four tools.
  An AI review is "a sensor, not a verdict".
• doubt-driven-development skill: give the reviewer the artifact and the
  contract, not the author's claims. Never re-run a review on an unchanged
  artifact. Stop after 3 cycles and escalate.
• code-review-and-quality skill: the cheap test check is one probe —
  "invert a condition the change adds and confirm a test fails". One probe,
  not one per test.

Matt Pocock

• code-review skill: two subagents with separate contexts, Standards (cites
  the file and rule) and Spec (quotes the spec line). Each brief is under 400
  words. It skips anything tooling enforces. It checks first that the diff is
  not empty, before it starts any agent. No Spec agent when there is no spec.
• Philosophy: invest in feedback loops (types, tests) and in agreement up
  front (grill-me, ADRs, agreed test seams). Then review at code time only
  checks against a contract. The agent must "produce proof that the code
  works".

Testing: what each source says

Checked a second time against the sources' own words.

Topic	Anthropic	OpenAI	Osmani	Pocock
Order	All tests first, commit them, then code "not to modify the tests"; a separate agent checks for overfitting (original best-practices post)	Little on test-writing. "Not letting the agent see its work" is a common mistake	Test first, "written without knowledge of the fix"	One test, then its code, repeated. The first is a tracer bullet. Writing all tests first is an anti-pattern ("bulk tests verify imagined behavior")
Refactor	Not covered	Not covered	Keeps red, green, refactor	Removed in June 2026: agents skipped it. Moved to code review, which carries a smell list
Strictness	"It is unacceptable to remove or edit tests" (long-running agents)	—	—	"stricter wording restricts the agent's creativity for little gain"
Where tests go	—	—	Unit-heavy pyramid; a separate test per case	Only at seams agreed up front; never internals, "no exceptions"
What review asks for	Flag only the gaps that affect correctness	Run code; fewer false alarms	Probe once per change by experiment	Check that only the agreed seams were used

#739 follows Pocock: build mode, tracer bullet, no refactor in the loop,
tautology rule. For refactoring, the last push's Fallow audit checks the
duplication and complexity the cycles added.

Others

• Cursor Bugbot: version 1 was 8 parallel passes with majority voting and
  a validator. Version 2 is one agent with tools that decides where to dig.
  That change gave "the largest gains": resolution rate went from 52% to over
  70%.
• Böckeler / Fowler, harness engineering: computational sensors (lint,
  types, tests) run on every change. LLM review is used selectively. Full
  mutation runs belong in the pipeline after integration.
• Simon Willison: ship a test that fails if the change is reverted, and
  paste the commands you ran with their output.
• Google, mutation testing at scale: mutate only the changed lines at
  review time, with a cap per line and per review, and skip lines that carry
  no logic. Developers found 85% of mutants useless at first; suppression
  rules cut that to 11%.
• Meta ACH: an LLM writes a few targeted mutants and then the tests that
  kill them; 73% of those tests were accepted. An LLM can detect equivalent
  mutants well (0.95 precision after simple preprocessing).

No source recommends agents mutating code by hand, test by test. The common
answer is a diff-scoped mutation tool, survivors fed back to the agent, and an
LLM to sort out equivalent mutants.

How we compare

Practice	Industry	Us	Gap
Deterministic gates before LLM review	Everyone	Strong: hooks, pnpm check, Fallow, comment-lint, stale-refs	None. Reviewers could be told more clearly to skip what these cover
Blind reviewer, fresh context	Anthropic, Osmani, Pocock	Yes: no PR body or commits, pinned inputs	None
Separate lenses	Pocock: 2 (standards, spec). Anthropic: bug finders plus rule checkers	3–4 (spec, code behaviour/tests, claims)	Spec and code overlap on test adequacy (calibration list shows duplicates)
Verify each finding	Everyone: one validator per finding	Yes: synthesis, then verify once per defect	None in shape. 484 rejected findings in 4 weeks shows the finders are noisy
Report bar	"If not certain, don't flag", P0/P1 only, ≤5 nits	Blocks need a failure or a named cost. Up to 5 notes	Notes still cost a verify and a write-up. Raise the bar to blocking only
Depth by risk and size	Anthropic, Osmani; trivial PRs get a light pass or are skipped	Model by risk (Opus vs Sonnet), but lens count by work kind, not risk	Small routine PRs still get 3 lenses
Test strength	Mutation tool scoped to the diff (Google, Stryker incremental, Meta)	Hand probes per test in audit (Sonnet) plus 4–9 probes in the Opus tests lens. The tool exists but only runs weekly	Biggest gap. Twice the work, done by hand, by the most expensive model
Loop limits	≤3 cycles; never re-review an unchanged artifact	1 full round + round 2 + fix-check, then park	Fine. Retries of fast "under 60 s" runs add hidden rounds (#683, #689)
Measurement	Addressed rate, false-positive rate, cost per review	Ledger designed, rarely filled	We cannot prove a saving
Repeated findings become lint	OpenAI harness, Anthropic best practices	comment-lint, stale-refs exist	Calibration list could feed new lint rules

Checks done more than once today

1. Mutation probes on new tests: implementer audit, then the Opus tests lens,
   then again after each fix run.
2. Test title vs body (rule 10): red stage, audit, tests lens, weekly scan.
3. Stale claims sweep: audit "Removed" step and the claims review.
4. Done-when / scope trace: audit "Promised" step and spec review Q1/Q3.
5. Rule breaches: tests lens Q2 and spec review.

Also a docs mismatch: REVIEW.md sends docs-only diffs to spec and claims, but
calibrate-reviews says only claims runs for them.

Proposed changes (ordered by saving / effort)

1. Diff-scoped mutation in delivery. After green, the implementer audit
   runs pnpm mutate <base> once, on its own diff. It kills survivors or marks
   them equivalent, with one line of reason each. The PR inputs get
   mutants.txt (survivors and decisions). The hand probe per test goes away.
   The tests lens no longer probes; it checks the survivor decisions and test
   shape. Guard rails: a cap on lines or mutants (the --max-lines flag
   already exists), a timeout, and results reused across fix runs (the cache
   already exists). Open question to measure: run time per typical PR. The
   weekly run takes about 15 min for 1500 lines; a 100–300 line PR should be a
   few minutes, but that is a guess.
2. Risk gate before lenses. routine + small (say under 150 changed
   source lines) + no ownership / persistence / concurrency / trust-boundary
   files → one combined Sonnet reviewer (spec + claims) plus CI and mutation.
   No Opus code lens. Everything else stays as today. This is the Anthropic
   plugin's Haiku-gate idea, done with our existing diff-kind.mjs and the
   risk reasons we already record.
3. One owner per check. The implementer audit owns rule 10 and the stale
   claims sweep (both mostly mechanical). Spec review owns Done-when and scope.
   The tests lens owns test logic only. Remove the copies from the other
   briefs.
4. Higher report bar. Adopt "if you are not certain it is real and it
   matters, do not report it". Drop notes from lens output (or keep notes only
   from the coordinator). Each finding: severity P0–P2, confidence, exact
   file:line, and a reproducer or quoted rule. Only P0/P1 go to verify.
5. Smaller handbacks. Lens reports capped at about 400 words plus the
   evidence lines; the evidence appendix stays (it fixed the fast-F0 runs) but
   as path:line refs, not pasted code.
6. Fill the ledger automatically. Read usage from the agent run results
   instead of asking agents to write it. Then compare 10 PRs before and after
   changes 1–4 on tokens, wall time, findings and escaped bugs.
7. Turn repeated rejected findings into lint. Go through the calibration
   list and the review history; any class that repeats and that a script can
   check becomes a rule in comment-lint, stale-refs or a new checker.

Not recommended: more parallel finders or voting. Bugbot moved away from
voting to one agent with tools and got better results. Anthropic says coding
splits into parallel parts poorly, and it costs about 15× the tokens.

Sources

• Anthropic Code Review blog: https://claude.com/blog/code-review (2026-03-09)
• Code Review docs: https://code.claude.com/docs/en/code-review
• /code-review local: https://code.claude.com/docs/en/code-review-local
• Ultrareview: https://code.claude.com/docs/en/ultrareview
• Code-review plugin prompt: https://github.com/anthropics/claude-code/blob/main/plugins/code-review/commands/code-review.md
• Security review action: https://github.com/anthropics/claude-code-security-review
• Best practices: https://code.claude.com/docs/en/best-practices
• Multi-agent research system: https://www.anthropic.com/engineering/multi-agent-research-system
• Building effective agents: https://www.anthropic.com/engineering/building-effective-agents
• Context engineering: https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
• OpenAI, Scaling code verification: https://alignment.openai.com/scaling-code-verification/ (2025-12-01)
• Codex on GitHub: https://learn.chatgpt.com/docs/third-party/github
• Codex SDK review cookbook: https://developers.openai.com/cookbook/examples/codex/build_code_review_with_codex_sdk
• OpenAI, Harness engineering: https://openai.com/index/harness-engineering/ (read through a proxy; the page returned 403)
• Osmani, Agentic Code Review: https://addyosmani.com/blog/agentic-code-review/ (2026-06-15)
• Osmani, Agentic Code Quality: https://addyosmani.com/blog/agentic-code-quality/ (2026-08-08)
• Osmani, agent-skills: https://github.com/addyosmani/agent-skills
• Pocock, skills: https://github.com/mattpocock/skills
• Pocock interview: https://newsletter.pragmaticengineer.com/p/ai-skills-with-matt-pocock (2026-09-17)
• Böckeler, Harness engineering: https://martinfowler.com/articles/exploring-gen-ai/harness-engineering.html (2026-04-02)
• Willison, Code proven to work: https://simonwillison.net/2025/Dec/18/code-proven-to-work/
• Cursor, Building a better Bugbot: https://cursor.com/blog/building-bugbot (2026-01-15)
• Meta ACH: https://arxiv.org/abs/2501.12862
• StrykerJS incremental: https://stryker-mutator.io/docs/stryker-js/incremental/
• Google, Practical Mutation Testing at Scale (2021): from secondary sources only

Not opened: Osmani's "70% problem", Kent Beck's own essay, Greptile and
Graphite primary docs (vendor numbers only through search results).

[V3RON]

Test redundancy, measured on src/leasing (2026-10-19)

A full Stryker run over src/leasing (2,643 mutants, every killing test recorded, 36 min) against its 627 unit tests:

Measure	Count
Mutants killed / timed out / survived / no coverage	1,672 / 555 / 319 / 97
Tests whose kills another test also makes, every one (subsumed)	228 (36%)
… by a test in the same file	213
… only by a test in another file (same claim at two seams)	15
Smallest set of tests that still kills every killed mutant	207 (33%)
Tests that kill nothing	28, mostly lease-request-book.test.ts, whose code sits in the registry, outside this run

What it says:

• Most redundancy is sibling tests in one file that split the cases of one behaviour (e.g. "41.7 is sent as 41" and "a percentage reported twice", identical kill sets). That is the pattern the new rule targets: a spec Tests line is one test, its cases a table. Merging keeps every case and drops the per-test setup.
• Same claim at two seams is small here (15), but real (lease-release-coordinator → create-leasing).
• Subsumed is not deletable: equal kill sets mean mutation cannot tell the tests apart, not that a case is worthless; the 555 timeouts have no killer recorded. Treat the list as candidates for merging, for audit-tests, not as a delete list.

Written by an agent.

[V3RON]

Mutation recall, replayed on 8 past weak-test findings (2026-10-19)

Would pnpm mutate (this PR's version) have caught the weak tests reviewers found by hand? Each run on the reviewed commit, cache cleared.

PR	Finding	Caught	Why not
#662 F1 (blocking)	removing a guard call left the test green	no	Stryker never deletes a call that takes arguments
#495 (blocking)	deleting await this.#removeRecord() stayed green	no	nor an awaited call
#586 (blocking)	moving a threshold 95 → 92 left 41 tests green	no	Stryker changes no number
#662 F2	an e2e check could pass with stay_awake never sent	no	e2e is out of reach
#537 note	a test sorts before comparing, hiding order	no	the weakness is in the test
#495 note, #651 note, #528 note	untested branch or log line	yes

2 of 8, none of the 3 blocking. Runs took under 1 min on small PRs, 5–9 min mid-size, 33 min on a 1,345-line PR, with 1–140 survivors unrelated to the finding.

So this PR keeps the hand probes (audit and tests lens) and adds pnpm mutate as extra evidence; it does not replace them. Next: a script that adds the two probes reviewers use most, deleting each changed call statement and moving each changed constant by one, replayed on the same 8 cases before any probe is removed.

Written by an agent.

[V3RON]

Testing changes, each measured before it was written (2026-10-19)

Build mode (Pocock-style), replayed on #491 in an isolated clone — one failing test: commit per Tests line, then its code:

	Original #535	Replay 1	Replay 2 (tightened brief)
Tests / test lines	10 / 153	10 / 145	10 / 160
One test per commit	no	no (5 in one)	yes, 10 test: commits
Later tests green on arrival	n/a	9 of 10	8 of 10, reported built ahead
Agent time / cost	unknown	222 s / $0.46	238 s / $0.43

The agent still writes the whole feature in the first green step; partly natural here (one matcher covers the later cases). Watch over the next 3 deliveries: share of built ahead, tests per Tests line (baseline 1.5–1.8: #618 13→20, #619 13→21, #626 stack 182→333), test lines, review test findings, build cost.

Rules:

Rule	Measured effect
"As today" behaviour proved by its existing test	2 of 107 tests in #631/#636 (2%); control cases of a new test group kept
Expected value never computed by the code under test (rule 10)	2 clear + 2 borderline in 150 sampled tests (1–3%); comparisons whose claim is "they match" allowed
Reviews ask for a test only when the spec, user docs or rule 3 require it, or for a defect the review found	32 of 77 review-driven tests (42%) not needed; 10 others guarded real bugs, hence the exception
Internal-file tests give a why: reason (pure logic, or an ordering the surface can't force)	leasing: of 267 internal tests, ~165 could move to createLeasing or go (estimate from a sample of 40)

Written by an agent.

[V3RON]

Mutation in delivery: dropped (2026-10-19)

A prototype added the probes reviewers use (delete a call, move a number by one) on changed lines, replayed on the 3 blocking weak-test findings:

Case	Result	Cost
#586 threshold 95 → 92	caught (95 → 94 survives, dies on the fix)	4 probes, 10 s
#495 deleting await this.#removeRecord()	caught (survives in 5 of 8 places)	106 probes, 913 s, 41 survivors
#662 F1	out of reach: the weakness was in the test's fake	34 probes, 761 s, 8 survivors

About 15 minutes a file and 20–40 survivors to explain, to catch what a hand probe catches in seconds. Mutation leaves delivery; the hand probes stay; the faster pnpm mutate stays for the weekly harden-tests routine.

Written by an agent.

[V3RON]

Benchmarks, round 3 (2026-10-19)

Each idea was measured on past PRs and transcripts before deciding.

Idea	Result	Decision
Cost and yield per lens (84 PRs, $368.70)	Round 1: $0.69 per fixed finding. Later rounds: $2.06, 36% of the cost. 17 later runs whose input the fix never touched found nothing	Shipped: a later round runs only the lenses its fix touched
Audit probe helper	It would save about 8% of audit cost. Agents already script their probes, and 85% of probe time is spent in vitest	Skipped. One line added: a break that doesn't apply must fail loudly (14 past breaks silently didn't apply)
Copied values in docs	31 fixed stale-claim findings were a copied value	Shipped documentation rule 8, plus the cleanup. An old-value check (3 catches, 4 false alarms) and a stale-refs.sh patch (2 catches) were skipped
Time waiting on tests	31% of implementer time	Shipped: the push hook's e2e typecheck goes from 15 s to 6 s. The 2 GiB download-cap test (128 s) is product code, to go in a separate task

Model replays (12 lens runs re-run on the other model)

Experiment	Recall of the fixed findings only this lens raised	Cost per run
Round-1 code/tests on Sonnet (#576, #562, #497)	5 of 13	$0.40–0.53, against $1.12–2.08
Later-round code lens on Sonnet (#693, #497, #567)	0 of 6 (#693's real bug came out only as a note)	$0.17–0.40, against $0.36–0.66
Spec on Sonnet, from Opus (#538, #462)	0 of 4	$0.13–0.40, against $0.39–1.03
Spec on Opus, from Sonnet (#735, #736, #738, #740)	About the same as Sonnet: one new finding, one missed	$0.52–0.94, against $0.14–0.45

Code lenses stay on Opus.

The spec lens stays on Sonnet. 3 of the 4 spec findings Sonnet missed were about test strength, and #739 moves that to the Opus code lens. On the four recent PRs, Opus added nothing for its higher cost. The one remaining miss was a design point: #462's refusal decided in two places. Watch for that kind of finding in the next deliveries.

Written by an agent.