Skip to content

chore(skills): harden sweep.py and close gaps in the rules - #1210

Merged
OmarAlJarrah merged 48 commits into
mainfrom
chore/skills-issues
Nov 1, 2026
Merged

OmarAlJarrah merged 48 commits into
mainfrom
chore/skills-issues

Conversation

@OmarAlJarrah

@OmarAlJarrah OmarAlJarrah commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

Summary

This resolves the skills: issues filed up to #1257 in one change, and #1230, a docs: issue on the same title budget the shipping rules state. Later ones, from #1262 on, stay open for a change of their own. Most of it is rule text. Four parts change behaviour: sweep.py, skills_lint.py, and two new tests.

#1152, #1155, #1185, #1198 and #1206 mention a skill but are about code, so they stay open. #1251 was closed as not planned, because the lint needs a checkout by design, so this PR leaves the lint's exit outside one as it was.

sweep.py. Two problems came from the same code, so they are fixed together.

  • A full disk ended a run in a traceback with no results. Compile output now stays in memory while it is compared, and only a spec that differs writes its outputs under OUTDIR/runs/<n>/. If those writes fail, the sweep carries on and says how many were not kept, because nothing it compares is on disk.
  • An IRVersion bump listed every spec that compiled. The version a document opens with is now set aside before comparing: the summary prints it once, a spec that differs in nothing else is stamp in results.tsv, and the listed specs are the ones that moved. A sweep with only stamp changes exits 0.

The direction in the issue was to delete each pair once it agreed. Holding output in memory goes further, because a compile writing to a full disk fails on its own write, which reads as a difference between the builds rather than a crash. On a full disk the old script recorded morphic: write ir document: write /dev/stdout: no space left on device beside documents cut short, before it stopped.

A run now also clears OUTDIR/runs, since rewriting only the specs that differ would leave an earlier run's pairs under numbers that can name another spec. An OUTDIR that cannot be made exits 2 with a message rather than a traceback, and a results.tsv that fails partway is removed rather than left to read as a sweep of fewer specs. An empty irVersion counts as a version, so it still shows as a difference.

skills_lint.py. The lint named a test as missing until its file was staged, so a rule naming a new test drew exit 1 from a run the curating skill asks for after every edit. It read test declarations from the tracked list and paths from the list that holds untracked files too. Both now read the one list, so an ignored or deleted file still declares nothing.

Review and merge against a stacked base. The review setup built the tree it reviews by merging main alone, so a stacked PR could pass on a tree that does not compile once it meets its base. The setup now reads baseRefName and merges that branch first. The same gap is closed at each other site: the gate pass, the new-commits pass, recheck-against-final-tree, the push checklist's conflict line, and a new merge-the-base-first rule. That rule is separate because adding this to merge-main-first took it past the word cap.

New or widened rules

  • trial-merge-open-prs now covers a change to what every file of a kind must hold, such as a golden, and runs that kind's tests on each merged tree. The review's process pass points at it. trial-merge-happened is new beside it: a trial merged from FETCH_HEAD in a new worktree fails, and its tests then pass on the branch alone.
  • trial-merge-lints-the-skills is new: a word cap is shared across open PRs, and this PR and feat(ir)!: settle doc references, promotion records and names #1219 merged with no conflict at 512 words against 500. This PR's additions to that skill are now short enough that the two come to 487 on today's main.
  • widened-field-values is new: when a lowering writes a value a field never held, read what the documents assume of the old values, and bump for the change of meaning, which no test sees. fix!: lower file bodies, enums, unions and streams as declared #1191's own 0.23.0 entry lists such a widening. The changing-the-IR description, its "What changes with it" list and the OpenAPI skill's traps all reach it, since nothing would otherwise have sent a reader there. widened-field-consumers is its second half, as a grep of the field and its old values finds nothing where a reader leaned on the old value unsaid.
  • cost-claims-per-arm is new: a cost through a function that switches on kind is measured per arm and quoted as a range. The issue allowed measuring only the arm that reads hardest. The rule asks for every arm, naming any left out, because the same reference opens by saying that reading the code does not find a cost.
  • limitation-searched-on-what-it-names is new: each limitation a body states is searched on the tool or behaviour it names, since a search on the construct can miss the issue tracking it. limitation-stated-where-it-lives and tracker-on-what-a-rule-settles both point at it.
  • new-id-namespace is new: a new entry in a compiler's Namespaces rewrites every golden document, where a version bump rewrites none, and needs a spec that mints it, since the compiler's tests hold the list to its corpus. The issue did not name that second part; a plant showed it.
  • rebuild-lost-plants is new, in mutation.md: a plant set nobody kept is rebuilt from the conditions the change adds rather than carried over, and run once before merge even after a comments-only merge. The review's code pass points at it.
  • verdict-outlives-unchanged-files is new, beside it: a plant at hand keeps its verdict across a merge that left its file and the test that killed it alone, so merge-main-first no longer reads as every plant after every merge. The issue's comment proposed the file alone, and said why that falls short: a merge can remove the test.
  • overlay-reaches-the-compiler-alone is new: go test -overlay plants without writing the tree, as that comment proposed, but it reaches only the compiler. A //nolint line and a new TypeKind each pass through an overlay and fail on disk, because internal/archtest and internal/decls read the sources at run time, so the rule says to re-plant such a survivor.
  • equivalent-means-every-input is new: a survivor is recorded as equivalent only with the reason no input can observe it, after compiling inputs across the construct's domain, since a green suite answers for the inputs it holds.
  • classify-named-strings is new, beside classify-ir-fields: a named string type needs an identityClasses entry. It is a rule of its own because classify-ir-fields is about fields.
  • mention-is-no-use is new: a TypeID position that names a type without using it stays out of ir.TypeEdges and of each reader that takes the class as a use. The issue asked for it to stay out of every reflective reader, which would be wrong for one resolving references, since a mention can dangle.
  • body-lands-at-merge is new: a commit body could stand on main when pushed, and a later commit making it stale is no finding, since a pushed branch is never rewritten and merges set their own body. A merge keeping the default paste edits a stale body out. commit-and-pr-message-form no longer claims the default squash decides what lands.
  • finish-checks-before-the-golden is new, split from tests-fail-fast-and-legibly: no assertion follows a golden write, and one that may start a parallel subtest runs inside a serial t.Run, since a guard ahead of the write sees neither failing. Its Probe plants a regression that changes the golden, as one that leaves the document alone keeps the golden with or without a guard.
  • deep-generated-shapes adds many $refs to one wide component, and reads output bytes beside instructions. The issue asked for the shape and the bytes; this also grows both dimensions together, since doubling either alone reads as linear. bound-the-product names references as a dimension.
  • build-guards-see-later-writes is new: a check on how a value is built also matches a later write that resets it, the forbidden value in each argument of an append, and a literal with an elided type. source-scanners-match-every-form gains every import of a path and a range clause's assignment, and its Probe a plant nested in another.
  • test-only-fix-reverts-by-hand is new: the revert that keeps test files is empty for a fix in test code, which read as a pass.
  • no-network-through-a-logging-proxy is new, with a row in the claim table. It states what the proxy cannot see, since a request to a loopback address and a client whose Transport sets no Proxy both leave the log empty.
  • plant-in-an-unpacked-copy and review-without-origin are new, for a repository that must stay unwritten or has no origin. The issue said git init for status and diff; the copy also takes a commit, since git init alone leaves them nothing to compare.
  • moved-prose is new, beside moved-comments: it covers text that moved word for word and went stale when its neighbour did not move, a citation that outlived its passage, and a rewrapped line that opens with block markup.

Found while making this change, so no issue tracks them

  • count-on-one-thread is new: an instruction count over a compile is taken under GOMAXPROCS=1 GOGC=off. Following instructions-not-wall-clock as written, the same two builds read +2.73% and then -2.08%; pinned, they read +2.06% with each side's range under 1%.
  • trial-merge-happened is new, as above: a loop here reported three green trial merges that had never happened.
  • pr-claims-match-evidence now holds a match count to the matches read, and a body edit to a push that has landed. A body here ran ahead of a commit the lint had refused, and a count of matching diagnostics turned out to hold ordinary spec text.
  • closing-keywords says to read the field seconds after a save. It listed one of this PR's issues straight after opening.

Corrected rules

  • rules-the-tree-still-breaks now names rules 6.15 and 6.24, and no longer implies its list is complete. test: most test doubles carry no stub, spy, fake or mock prefix #1279 tracks 6.24, as no issue did.
  • redundant-guard asks whether removing a guard changes any outcome. A guard whose removal changes none is deleted, where the old text told a review to pin something no test could fail on. CLAUDE.md's sentence on guards stays as it is: it already gives this answer, and the rule now agrees with it.
  • better-than-main no longer sends every output change to a Breaking section, which morphic-shipping-prs ties to a ! in the title.
  • mutation-survivor-scope no longer asks for a whole-module run for a helper in a _test.go file, where its directory and ./internal/archtest settle it. This departs from the issue twice. Its own directory is not enough, since archtest reads test files as text. And export_test.go symbols are not an exception: another package importing the package does not compile that file.
  • The verifying skill's habit lets a review's gate pass take CI's run for go test ./... too, as it already does for make gate. Pass 1 says a head behind a moved main gets no such exception.
  • one-mutant-per-verdict says a plant removing a call removes it, as one discarding its error still runs it, and mutation-survivor-scope says an arm's removal alone does not make it equivalent.
  • CLAUDE.md counts the appended number at the number the PR will get. It fixed the width at three digits, so a title its command passed at 72 landed at 73.
  • CLAUDE.md and the shipping skill said a PR lands under its title. A branch with one commit of its own lands under that commit's subject, as CONTRIBUTING.md already said and test(ir/irverify): make the sort test reach its Path tiebreak #1232 shows, so every commit subject now counts the number, and the push checklist measures each as it would land.
  • The IR skill's hand lists line says which list each kind of field meets. It said which test, and the reference names no test per table.

A departure to weigh. #1246 left two ways to settle assert.Equal against rule 6.16, which takes cmp for slices and maps and bars reflect.DeepEqual, which assert.Equal calls. assert-equal-on-plain-values records a departure for a value holding no unexported field, and CLAUDE.md says the same; a value holding one keeps cmp.Diff. The rule's reason is unexported fields. On a plain value the two agree on nil against empty, on order and on a NaN in separate values, and cmp parts from assert.Equal in two probed cases: it calls an Equal method, so the departure leaves out a value declaring one, which no type here does, and it refuses a NaN in a slice compared with itself. The tree compares a literal slice or map with assert.Equal about twice as often as it calls cmp.Diff. The alternative is a row in rules-the-tree-still-breaks and an issue to convert the calls.

New checks. TestAppendTypeIDs_ReadsWhatTheTypeGraphReads fills every kind and holds appendTypeIDs, which reads every TypeID under an operation by reflection, to ir.TypeEdges. A position that only mentions a type would otherwise become a GraphQL reachability root, and the check on field arguments would go silent on that type, with nothing saying so: the tests that fail on a new position ask for it to be classified, which says nothing about this reader. The test keeps no list, so it passes on #1219's DocRef position with no edit.

TestDocuments_OpenNoLineWithAPlus refuses a line opening with + outside a fence, across the repository's documents and .claude/skills, which a walk of the root leaves out. No document writes a list that way, so such a line is a sum a reflow left at the start of a line, and GitHub renders the rest of the paragraph as a list.

Limitations

  • TestAppendTypeIDs_ReadsWhatTheTypeGraphReads reaches a type definition's fields. A position only an operation holds is no field of a kind, so nothing holds the reader to it: such a field added to ir.Operation leaves go test ./pass green, which mention-is-no-use states.
  • TestDocuments_OpenNoLineWithAPlus holds the plus marker alone. Every other marker has real lists to tell it from, so moved-prose leaves those to a Probe that prints candidates to read.
  • The push checklist's conflict line reports conflicts, not failures, as trial-merge-open-prs says. A base that merges cleanly but breaks the build is caught by merge-the-base-first and the review setup, not by the checklist.
  • sweep.py holds each spec's output in memory for as long as it is compared, so -j bounds what it holds.

A tracker search on each of these (appendTypeIDs, TypeEdges, reflective reader, list marker, markdown block markup, merge-tree, push checklist, sweep memory, sweep.py) returned only issues this PR closes or ones on other subjects, so no issue tracks them.

Closes #1184
Closes #1186
Closes #1189
Closes #1193
Closes #1196
Closes #1197
Closes #1203
Closes #1204
Closes #1205
Closes #1208
Closes #1209
Closes #1213
Closes #1216
Closes #1223
Closes #1224
Closes #1225
Closes #1230
Closes #1236
Closes #1237
Closes #1242
Closes #1243
Closes #1246
Closes #1252
Closes #1253
Closes #1254
Closes #1255
Closes #1256
Closes #1257

Test plan

Sweep behaviour, on real builds of cmd/morphic at this head's sweep.py:

  • Version bump: fix!: lower file bodies, enums, unions and streams as declared #1191's head 2c2404be against 20017d99, over testdata and a corpus of real specs, 2,170 specs in all. The old script lists 2,075 and this one 174, with 1,901 marked stamp. An independent comparison of the old script's kept outputs, as JSON with irVersion removed, agrees with every one of the 2,170 verdicts.
  • No bump, real differences: test: witness common formats and pin what the suite left unheld #1192's head 46125b1b against bd874baa, over testdata alone, 215 specs. results.tsv is byte-identical to the old script's, each of the 212 kept pairs matches its bytes, and the printed rows match. The one printed change is the header, which no longer names an outputs directory.
  • Full disk, on a 130 MB RAM disk over the same bump. The old script ends partway, at run 550 and at run 761 in two attempts, with a traceback and no results.tsv. This one finishes all 2,170 specs and reports the pairs it could not keep, 112 and 113 in those attempts, and none of its verdicts differ from a run with room.
  • python3.14 -m unittest discover passes for both script directories, eight concurrent runs of the sweep's suite included, and 36 runs of its command-line tests, twelve at a time on a loaded machine. 32 mutants of sweep.py are each killed by a test, and a comment-only control survives.

Evidence the rules cite, replayed rather than quoted:

  • 0611ef9b passes go vet ./internal/archtest alone and fails once merged with d707412d. On a local remote holding that stack, with main at 20017d99, the old setup stops on four conflicts with the head alone, where the failure cannot show. The new one merges the base first, which is the merge that fails, and then stops on the conflicts with main, leaving that tree to review.
  • 21854da9 and ba2f961c merge with no conflict, and TestConformance fails on the merge.
  • At 46125b1b, replacing the getter with a field read panics TestAllOf_BoolBranchContributesNoRequiredNames, and deleting the neighbouring bs == nil guard panics the same test at unmergedBranchKeys.
  • Per-format cost at e67594b2 against 20017d99, instructions retired under GOMAXPROCS=1 GOGC=off, three alternating runs a side on 65,000-member specs I generated: double adds about 34%, int64 about 2% and uuid under 1%, with each side's range far narrower than the gap on double. That is the spread the rule describes; the figures in its Why are the ones its citation pins.

The rest:

The lint named a test as missing until its file was staged, since it
read test declarations from the tracked list while reading paths from
the list that holds untracked files too. A rule naming a new test drew
exit 1 from a run the curating skill asks for after every edit. Both
now read the one list, so an ignored or deleted file still declares
nothing.
appendTypeIDs reads every TypeID under an operation by reflection, so a
position that names a type without using it would become a GraphQL
reachability root, and the check on field arguments would stay silent on
that type. The tests that fail on a new position ask for it to be
classified, which says nothing about this reader.

The new test fills every kind and compares what the reader takes with
ir.TypeEdges. It keeps no list, so a position left out of the graph
reaches it unaided. A position only an operation holds is no field of a
kind and stays out of its reach, which the new rule states.
classify-ir-fields lists what a new field meets, but a new named string
type meets identityClasses, which it does not name. The test finds out,
while a change sized from the rule does not. The rule is about fields
and at its word cap, so this is a rule of its own beside it.
A mutant of a helper in a _test.go file can be seen only by its own
directory's tests and by archtest, which reads test files as text, so a
whole-module run per survivor repeats what those two settle. That holds
for export_test.go too: another package importing the package does not
compile it. A review whose gate pass took CI's run also need not repeat
it to conclude.
The message rule required every pushed body to stand on main because
the default squash pastes them all, but merges set their own body and a
pushed branch is never rewritten, so a body a later commit made stale
could never be fixed and a review could not tell whether to report it.
A new rule holds a body when it is pushed, and the body that lands at
merge.

CLAUDE.md measured the appended number as three digits. Numbers are
four digits now, so a title its command passed at 72 landed at 73. It
now counts the number the PR will get, as the push checklist does.
Added directly under the hand lists bullet, the goldens bullet sat
against that bullet's last line, which #1238 rewrites, so the two
conflicted on nothing but position. One bullet down, they merge. A
shorter values bullet keeps the body within its cap once #1219 lands
too, which together with this branch's additions it would pass.
The rule said to stop before a golden write when an earlier assertion
failed, which a runner can follow and still rewrite a golden from
regressed output: an assertion after the write has not run yet, and a
parallel subtest runs only after its parent returns, so at the guard
nothing has failed. The rule now states what a run does, and its Probe
plants a regression that changes the golden, since one that leaves the
document alone keeps the golden with or without a guard. It is a rule of
its own, as the rule it came from was over its word cap with this added.
The generated shapes left out many references to one wide component, and
the Probe read instructions alone, so a product of references and a
component's size passed them. The rule adds the shape, grown in both
dimensions together, since doubling either alone reads as linear: output
grew 2.00 times with references and 1.95 with entries, against 3.89 with
both. The Probe reads output bytes too.
Rule 6.16 takes cmp for slices and maps and bars reflect.DeepEqual,
which assert.Equal calls, while the tree writes assert.Equal on them
about twice as often as it calls cmp.Diff, so a review could not tell a
breach from the house spelling. The rule's reason is unexported fields,
and on a value with none the two agree, on nil against empty, order and
NaN alike, while assert.Equal labels both sides of a failure. This
records a departure for such a value rather than a row awaiting the
conversion of every call, and CLAUDE.md says the same.
Outside a checkout the lint printed the git command that failed, which
names no problem. It now says the tree is read from git and how to get
one. It does not fall back to walking the disk: a copy of .claude alone
holds no module, so every path check there would be a false finding,
while git init in a full copy already lints clean.
Each of these reached a review of #1233, #1235 or #1238 as something the
reviewer had to work out alone.

build-guards-see-later-writes is new: a guard on how a value is built
also matches what undoes the build, since one passed a later write, a
later append argument and an elided literal in turn. one-mutant-per-verdict
says a plant removing a call removes it, as one discarding its error
still runs it. mutation-survivor-scope says an arm's removal alone does
not make it equivalent. test-only-fix-reverts-by-hand is new: the revert
that keeps test files is empty for a fix in test code, which read as a
pass.

no-network-through-a-logging-proxy is new, with a row in the claim table.
An empty log says nothing about a loopback address or a client whose
Transport sets no Proxy, and the rule says so. plant-in-an-unpacked-copy
and review-without-origin are new, for a repository that must stay
unwritten or has no origin. The copy takes a commit, since git init alone
leaves status and diff HEAD nothing to compare.
A word cap is shared across open PRs. Two branches each under a skill's
body cap merged with no conflict at 512 words against 500, which neither
a conflict check nor the tests show, so the second to land would fail
the lint on nothing in its own change.
The lint reads the tree from git by design, and exit 2 outside a
checkout is the "cannot run" its docstring defines, so #1251 was closed
as not planned. Saying more on that path answers nothing anyone asked
for, so this takes the message back out.
A sweep of two builds at one version had no test, so reporting every
spec as a stamp change passed. Nothing pinned two pairs of versions in
one run or a stamp that is not UTF-8 either.

The killed-compile test raced a one-second limit against an interpreter
starting, which its neighbour's comment rules out, and it failed on a
loaded machine. It now raises the timeout itself, and also covers a side
that wrote nothing, which the library may leave as None.

missing_ok was dead under suppress(OSError), and the summary now says
how many of the specs a stamp change covers.
cmp calls an Equal method and assert.Equal does not, and the two part on
a NaN when a slice is compared with itself, so "agree on NaN" claimed too
much and the departure reached a value it should not. No type here
declares Equal, which the Probe now shows. The count it cites is of a
literal slice or map, not of every assert.Equal, so it says that.
Two rules said "above" of a neighbour, which a move leaves pointing at
nothing, and one Probe's "that test" could mean either of two. Each now
names what it means. The rest is rewrapping, bar one Why reworded.
The title rule said a PR lands under its title, but the repository's
squash setting lands a branch with one commit of its own under that
commit's subject, and a merge of main is no second commit. #1232's
61-character title landed as a 73-character subject that way.
CONTRIBUTING.md already said so for Dependabot; CLAUDE.md, the shipping
skill and its message rule did not.

The push checklist now measures each subject as it would land.
merge-main-first read as every plant after every merge, which on one
PR came to about 3.5 hours for a merge that changed no lowering. A
verdict now outlives a merge that left its file and its killing test
alone.

Planting through go test -overlay writes no tree and runs in parallel,
but only the compiler sees it. A plant caught by reading sources, such
as a //nolint line or a new TypeKind, passes through an overlay and
fails on disk, so the rule says to re-plant such a survivor.

Equivalence is a claim about every input, and a green suite answers
only for the inputs it holds: deleting a guard in allOf's election
passed go test ./... while one order of a two-conjunct allOf lowered
to any and the other to integer.
source-scanners-match-every-form gains every import of a path and a
range clause's assignment, and its Probe a plant nested in another, each
replayed against the child-command rule at the parent of its fix.

Pass 1 says a green head behind a moved main gets no exception. A
widened field's readers get a rule of their own, since a grep of the
field and its old values finds nothing to change when a reader leaned
on an old value unsaid. rules-the-tree-still-breaks names rule 6.24, and
the IR skill's hand lists line says what its reference gives, which is
a list per kind of field and not a test.
load_tree reads declared test names only from files ending in _test.go,
and nothing held that: reading every .go file passed the suite, while a
skill could then name a function no test file declares. A non-test file
declaring one now sits in the fixture.
@OmarAlJarrah
OmarAlJarrah merged commit 608a5a4 into main Nov 1, 2026
2 checks passed
@OmarAlJarrah
OmarAlJarrah deleted the chore/skills-issues branch November 1, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment