Repository navigation
ci: move the fuzz search from pull requests to a nightly job - #1301
OmarAlJarrah wants to merge 15 commits into
Conversation
go test -fuzz instruments everything a test binary links, and every execution clears, copies and scans the coverage counters of all of it. A binary links every test file's imports, so ir's fuzz targets were paying for golang.org/x/tools through internal/decls, and FuzzCompile for the end-to-end suite's test tooling. Coverage bookkeeping took 52% of a FuzzCanonicalWords worker's CPU and 33% of a FuzzCompile worker's. Move ir's five targets to ir/fuzz and the three that compile whole documents to compilers/openapi/e2e/fuzz, test-only packages that import no test tooling. The tables ir's targets share with table tests move to internal/ircases, which imports only the standard library, and the ref-shaped specs to internal/testspec, so a row added for one still reaches the other. FuzzCycleDetector now seeds from the whole fixture directory, which also stops a read error costing it its seeds in silence. CONTRIBUTING's rule that a test sits beside its code gains the exception and when it applies. Coverage counts a package's own tests, so nothing moved stands in for them: ir's statements stay covered with these seeds gone.
Every pull request spent a median 231s on a search that restarted from the seeds each time. Pull requests now run make fuzz-check, which holds the targets to their cap and the quarantine to real targets without searching, and every target's seeds still run in the suite. Fuzzing leaves make gate. The search runs on a schedule or by hand, in a job of its own, with a corpus it restores and saves between nights under a key nothing else reads. Go never trims that directory, so make fuzz-prune drops targets that are gone, entries over 64 KiB, and anything past 1,000 entries a target. A failed search uploads its findings and comments on one standing issue. That job alone may write to issues, only the reporting step holds the token, and the checkout leaves no credential. fuzz.sh reads its targets from the packages ./... names, so a plant under testdata no longer spends the budget or trips the cap, and it searches every target after a failure, so one finding cannot cost the rest of the night. Each search minimizes for 1s, not 10s: the flag also governs every input that adds coverage, and a worker minimizing runs no search. At one budget that found 20% more inputs in the same time. A search run by hand defaults to 5s. The lint step sets the -trimpath the Makefile exports for every other step, so it can reuse what make vet compiled.
go test handles an interrupt and exits, which the search reads as a failed target and carries on from, so Ctrl-C stopped one target of fifteen. Before the search learned to carry on past a failure, set -e stopped it at once. Trap the interrupt so the search exits 130 once go test has gone.
The corpus shares the repository's 10 GiB cache limit with what main saves, at about 1.1 GiB a push, and eviction takes what was used longest ago. Enough pushes in a day lose it, and the search then starts from the seeds, which is what it did before it kept a corpus. That costs depth and nothing else, so it should not fail the run, but it should not pass unnoticed either.
CLAUDE.md's testing rule asks for testify, which the fuzz packages exist to leave out, so it now names the exception. CONTRIBUTING names internal/testspec beside internal/ircases, since the end-to-end targets read their table from it. A narrower edit to docs/ir/naming.md and a different place for ircases' archtest entry keep clear of lines open pull requests also change.
prune decides between keeping a target and rm -rf with printf piped to grep -q. Under pipefail, grep leaving early kills printf, the pipeline reports failure, and a live target reads as gone. The list is small enough today that printf finishes first, but a larger one missed its first line 5 times in 5. A here-string has no writer to kill.
1,000 entries a target was a guess, and five simulated nights refuted it: FuzzCycleDetector reached 908 with its coverage still rising, from 42,394 bits to 48,137, so the cap would have started evicting at random around night six. Its gains fell by about a fifth a night (264, 220, 168, 145, 111), which points to roughly 1,400 entries at that search rate. A search re-runs every entry inside its budget, 9s at 900 entries under load, so the cap cannot rise freely either; 3,000 leaves twice that margin. A target past it now says so in the log.
docs/architecture.md's package tree had no entry for ir/fuzz, the end-to-end fuzz package or internal/ircases, and CONTRIBUTING still listed fuzzing among what goes in compilers/openapi/e2e.
internal/ircases and internal/testspec exported their tables as package-level slices, so any test could change a row every other test in its binary reads. CLAUDE.md bars mutable globals, and the styleguide bars handing out a shared slice (5.29) or mutable package state (5.30). No test-support package exported a variable before them. Each table is now a function returning a fresh slice, and a test per function holds that: it edits the rows it was handed and reads them again. A function is a statement coverage counts, so testspec gains its first test, and archtest's testSupport its entry, since only tests reach RefShapedData.
The checks it made through testify's assert became t.Fatalf when it moved, which the styleguide reserves for a failure that makes going on meaningless (6.14). Both checks end the input anyway, so t.Errorf reports the same.
fuzz.sh takes check, prune or a -fuzztime budget as its one argument. A mistyped mode was handed to go test as a budget, so every target built before rejecting it: chek took 240s to fail. It is now refused at once, with exit 2.
The comment said only the reporting step held the token. setup-go is handed it too, as its token input's default, so that was not so. The job's safety rests on which code it runs: main's, or a writer's dispatch, never a pull request's.
ir/fuzz and compilers/openapi/e2e/fuzz sit apart from their code, assert without testify and read tables from a package not named for ir, against rules 6.20, 6.5 and 2.9, because a fuzz run pays for everything its binary links. Recording it keeps a style review from reporting it, and names #1315, the check that would hold the packages to linking no test tooling.
docs/ir/naming.md keeps both sides: this branch says the fuzz target is seeded from the conformance table, and main adds that the table's answers are held to IRVersion. ir/irversion_names_test.go, from main, named canonicalCases, which this branch moved to internal/ircases as Canonical; its comment now says so.
As a function returning a literal, RefShapedData ran to 78 lines, past the 70-line cap funlen holds production code to, and Canonical would cross it once its table grew. Each table is now an unexported variable, and the exported function returns slices.Clone of it: a fresh slice for every caller, as before, in a function that stays one line however many rows the table holds.
fuad-daoud
left a comment
There was a problem hiding this comment.
Reviewed e982961f, and that head merged with main (b206dc99, clean). Two issue:s and three nit:s. The rest of #1273's Direction is in place and checks out; the item-by-item table is at the end.
Gate. It passes on the merged tree:
Coverage gate passed: all 17377 statements covered.fuzz-check:15 fuzz target(s), 15 searchable.bench-smokepasses.
The first attempt failed only in canary and load, on timing, while the machine was under load, and both pass alone (33s and 85s). CI is green on the head.
issue: the body says prune keeps 1,000 entries a target; the code keeps 3,000
c6e0265d raised max_corpus_entries to 3000 (scripts/fuzz.sh:72). Both the Summary and the Test plan still say 1,000.
Run on a fake GOCACHE holding 3,005 entries for a live target, the old ir/ directory of a moved target, and entries of 65,536 B, 65,537 B and a few bytes:
fuzz.sh: FuzzCanonicalWords_Properties is past 3000 entries; dropped 5
It kept 3,000, deleted the moved target's directory, and kept the 65,536 B entry while dropping the 65,537 B one.
Direction: say 3,000 in both places, and rerun the Test plan's prune item at that cap.
issue: two changes here are their own pull requests
#1273's Direction is F2–F5 plus -trimpath, and its Out of scope names -fuzzminimizetime (#1268's F6: measure it on the nightly job). Two groups of hunks trace to no item of it:
-
The target moves. This covers:
ir/fuzzandcompilers/openapi/e2e/fuzz;internal/ircases;- the new table in
internal/testspec; - CONTRIBUTING's fuzz-package rule, CLAUDE.md's testing exception, and
departures.md#fuzz-packages-link-nothing.
The nightly job and
fuzz-checkwork the same with the targets where they are onmain, and the body already gives this its own section. The gain is real: re-measured onFuzzNewBigVal_AcceptedFormsAreJSONValid, 200,000 executions, alternating, CPU fromgetrusage, the median went from 233 µs to 86 µs (63% saved, against the body's 71%). It reads better as its own pull request, with #1315 beside it. -
minimize_time10s → 1s, which closes #1290. It has its own issue and measurement.
What stays with #1273, since each traces to it:
- the interrupt trap fixes this branch's own carry-on loop: with the trap removed, a SIGINT left 3 targets started 8s later;
- the argument check guards the new
checkandprunemodes; - searching every target after a failure is what the nightly report needs;
- the first
departures.mdhunk is a site of "fuzzing leavesmake gate".
How we'll take it from here. So this doesn't wait on a rework, we'll carry it as three pull requests built from your commits, with you as their author, as #1313 did with #1264:
- #1273 itself (closes #1273), stacked on #1311 and #1272, since all three change the same
gate.ymlandMakefilelines; - the target moves, with #1315 beside them;
-fuzzminimizetime(closes #1290).
The issue above and the nits below get fixed there. Once the first of these opens, I'll close this pull request with a link to all three.
nit: fuzz.sh refuses budgets go test -fuzztime accepts
The comment at scripts/fuzz.sh:23-26 says it reads the budget "as go test reads -fuzztime". The script refuses .5s, 1.s and 5µs with exit 2, while go1.27.2's -fuzztime accepts all three. On main the script handed the budget straight through.
Direction: ^([0-9]+x|(([0-9]+(\.[0-9]*)?|\.[0-9]+)(ns|us|µs|μs|ms|s|m|h))+)$.
- It accepts
5s 1m30s 0.5s .5s 1.s 5µs 5μs 100x 1h2m3.5s. - It refuses
chek 0 -5s . .s 5 x 1.5x(bash 5, and underLC_ALL=C).
nit: FuzzCycleDetector's doc says it is "seeded from the cycle corpus"
compilers/openapi/e2e/fuzz/cycles_test.go:15. It now seeds all 66 .yaml files in testdata/openapi: 79 seeds, against 44 on main. One of them, tagged_mapping_request_body.yaml, is not a cycle fixture.
Direction: "compiles arbitrary bytes seeded from every fixture in testdata/openapi and the ref-shaped specs".
nit: CONTRIBUTING's new rule does not say that nothing holds a fuzz package to it
Its Fails if missed names two outcomes. It leaves out the rule's central one: an import of test tooling in a fuzz package fails nothing yet (#1315). departures.md and the body say so, but CONTRIBUTING is where a contributor adding a target reads.
This one goes with the target moves.
#466
The decision to give a deadline no handling holds:
- Source. go1.27.2's
src/internal/fuzz/fuzz.go:129-137drops the error whenerr == ctx.Err() || err == fuzzCtx.Err(), citing go.dev/issue/75804. That line first appears in 1.27.1: 1.22.12, 1.26.3 and 1.26.6 lack it. - Timeline. The repository moved to 1.27.1 in
20778b20(#471, 2026-09-21), and both of #466's occurrences (09-12 and 09-13) came before that. - CI history. 37 failed gate runs since 09-20: 13 reached the fuzz step, and none failed it.
I've added the source lines and the timeline to #466.
Against #1272 (C1 + M1, opening soon, stacked on #1311)
The two coexist:
- M1's restore. It fits, since your
fuzzjob restores only$GOCACHE/fuzzunder its own key. - #1272's
drop the fuzz corpusstep becomes a no-op oncemain's pushes never pass-fuzz.
One hazard for whichever lands second: taking this side of the checks conflict deletes the &drop-fuzz anchor while suite keeps *drop-fuzz, and actionlint then reports unknown anchor 'drop-fuzz', so the whole workflow fails to parse. The #1273 pull request above will remove the step and both aliases, and correct #1272's comments that say the corpus has no cache of its own yet. Against #1311, the resolution in the body holds: actionlint is clean, and make -n gate runs race, then fuzz-check.
#1273's Direction, item by item
| Item | Verdict |
|---|---|
F2: targets from go list ./... |
Present and correct. A FuzzPlanted under internal/lintrules/testdata/zz is not counted, where main's git grep counts it. |
| F2: cap, stale-quarantine and zero-target checks on pull requests | Present and correct. 16 targets pass, 17 refuse; a stale quarantine, every target quarantined, and a changed grammar each refuse. |
F2: fuzzing out of make gate |
Present and correct. No prose still says the gate searches. |
F3: schedule + workflow_dispatch |
Present. Neither trigger has run, as the body says. |
| F3: a job of its own, and its own concurrency group | Present and correct. |
| F3: within the 30-minute timeout | Fits by estimate: about 18–20 min at FUZZTIME=1m and the 16-target cap, from the trial runs' build time. |
F4: $GOCACHE/fuzz only, under a prefix PRs don't restore, with run_id in the key |
Present. run_attempt in the key is an improvement. |
| F4: prune of gone targets, size cap, save on failure | Present. The cap is 3,000, not 1,000 (first issue). |
F5: upload, comment on #1296, issues: write on the job only |
Present and correct, with persist-credentials: false. |
-trimpath on lint |
Present and correct. |
| CLAUDE.md's two-job sentence | Present and correct. |
|
Carried forward as three pull requests built from your commits, with you as their author, as the review above said:
Closing this one in favour of those. Your branch |
The departure's Why names darwin/arm64 for its counter figures, and neither #1301 nor #1315 records a platform. The check is on #1381: cross-built for darwin/arm64, ir's binary at 608a5a4 counts 64,256 and ir/fuzz's at 113c9e9 counts 14,366, as cited; linux/amd64 counts 64,701 and 15,112 at the same revisions.
Summary
Every pull request spends a median 231s on a fuzz search (the 79 runs in #1273) that restarts from the seeds each time, so the search never adds up. This moves it to a nightly job that keeps its corpus, and makes each execution of it cheaper.
Closes #1273.
Closes #1290.
The search leaves pull requests
make fuzz-checkreplaces the search in the gate. It holds the targets to the cap, the quarantine to real targets and the list to at least one searchable target, and searches nothing. Every target's seeds still run ingo test ./.... Fuzzing leavesmake gate.fuzz search, runs on a schedule or by hand and never beside the other two. It has its own concurrency group, since a scheduled run carriesmain's sha and would otherwise sharegate-<sha>withmain's push run. At 1m a target it stays inside the 30-minute job timeout at the target cap.$GOCACHE/fuzzalone, under afuzz-corpus-key no other job restores. Go never trims that directory, somake fuzz-prunedrops targets that no longer exist and entries over 64 KiB, and keeps 1,000 entries a target. This is a deliberate exception to "the cache never grows" (ci: run the gate as two jobs and drop the Windows and wasm ports #1025, ci: carry main's Go build cache into every run, update actions #755).issues: write, and it runs onlymain's code or a writer's dispatch, never a pull request's. The checkout leaves no credential in.git/configand the search step's environment holds no token, thoughsetup-go, as itstokeninput's default, and the reporting step are both handed one.fuzz.shreads its targets from the packages./...names, so a plant undertestdatano longer spends the budget or trips the cap. It also searches every target after a failure, because one finding should not cost the rest of the night's search. An argument that is neithercheck,prune, a duration nor a count is refused before anything builds, where a mistyped mode used to build all 15 targets first.lintstep setsGOFLAGS: -trimpath, as theMakefiledoes for every other step, so it can reuse whatmake vetcompiled.The
lint, fuzz, benchjob is nowlint, bench. No ruleset requires a check by that name (gh api repos/dexpace/morphic/rulesets/19179519 --jq '[.rules[].type]').Each execution costs less
go test -fuzzinstruments everything a test binary links, every execution clears, copies and scans all of those counters, and a binary links every test file's imports. Coverage bookkeeping took 52% of aFuzzCanonicalWordsworker's CPU and 33% of aFuzzCompileworker's, against 5.5% and 1.1% in Morphic's own frames.So eight targets move to packages that import no test tooling:
ir's five toir/fuzz, and the three that compile whole documents tocompilers/openapi/e2e/fuzz. The tables they share with table tests move tointernal/ircasesandinternal/testspec, because each one's comment says a row must not go missing from the seeds. Each is an unexported variable that a function hands out a copy of, so no test can change what another reads; the tables had been unexported variables of one test package each, and exporting the slices themselves would have brokenCLAUDE.md's "no mutable globals".CONTRIBUTING.md's rule that a test sits beside its code gains the exception and when it applies, andmorphic-checking-style'sdepartures.mdrecords it against the styleguide's rules 6.20, 6.5 and 2.9 (fuzz-packages-link-nothing). No check holds the fuzz packages to linking no test tooling yet; #1315 proposes one.CPU per execution, one worker, fixed executions, old and new binaries alternating, five runs each from
608a5a46under heavy machine load:FuzzCanonicalWords_PropertiesFuzzHasCanonicalWords_AgreesWithCanonicalWordsFuzzNewBigVal_AcceptedFormsAreJSONValidFuzzParseDecimal_ReadsWhatNewBigValKeepsFuzzDecimal_OrdersAsRationalsDoFuzzLowerSchemaFuzzCompileFuzzCycleDetectorEach binary dropped coverage counters:
ir's from 64,256 to 14,366, ande2e's from 164,553 to 95,479 beforeFuzzCycleDetectorjoined it.FuzzCycleDetector's two sides start from different seeds, since it now reads the whole fixture directory, so its row is the least exact. Finds grow more slowly than executions: in three pairs of 10s searches of each of three targets on6bbbc8f8, 1.5–3.4× the executions gave 14–23% more new inputs.Decisions worth reading
-fuzzminimizetimegoes from 10s to 1s. ci: move the fuzz search from pull requests to a nightly run #1273 left this out of scope and ci: set -fuzzminimizetime for the nightly search by the inputs it finds #1290 asks for it to be measured at the nightly budget, by what the corpus covers. The flag governs every input that adds coverage, not only a crash, and a worker minimizing runs no search. Fresh corpus each run, three runs a side, alternating,-fuzztime=1m -parallel 4, with coverage read from the corpus throughGODEBUG=fuzzdebug=1, which reads 410 bits for a target's seeds alone against 854 after an 8s search:10s1s10s1sFuzzCycleDetectorFuzzNewBigVal_AcceptedFormsAreJSONValidThe cost ci: set -fuzzminimizetime for the nightly search by the inputs it finds #1290 names did not appear: over four runs a side on a planted crash, reproducers were 2 or 3 bytes at both bounds, and 10s missed the crash once within 2 minutes. This compares one shorter bound on two targets. At a 10s budget across all 15 targets, 1s also found 1,035 new inputs against 863, and a search run by hand now defaults to 5s, which found 812.
A deadline with no finding gets no handling of its own. The pinned Go suppresses it (
internal/fuzz/fuzz.go, go.dev/issue/75804), and none of 210 searches here failed on one at a load above 140. None of the 36 failed gate runs since 2026-09-20 shows a failed fuzz step, which is weaker than it sounds: only 13 reached the step, and a run re-run to green would not appear. So a red night is a finding or a failure worth reading, and ci: fuzz step reddens on the coordinator's own deadline with no finding #466 can be judged on that evidence separately.Seven targets stay beside their code.
loadandoperationneed their package's internals.navigationandnodeviewcompare against the parser library, so they keep its code.barrier,compileandfragmenthold about 29,000 counters already, so there is little to shed. None of these seven was measured in a package of its own.Running targets side by side was tried and left out. Each target already uses every core and
-fuzztimeis wall clock, so four at once with one worker each did the same executions as 2.5s one after another with 13% fewer finds. It saved startup and shutdown, and would have cost readable logs and Ctrl-C.What this gives up: a bug only a search would find now reaches
mainand waits for the night, where a 10s search on the pull request might have caught it. ci: cut a pull request's gate from a 9-minute median to about 3.5 #1268 weighs that trade.The corpus can be evicted. It shares the repository's 10 GiB cache limit with what
mainsaves, about 1.1 GiB a push, and eviction takes what was used longest ago. At608a5a46the limit held about nine pushes, 28 hours back, andmaintook seven in the last day (gh api 'repos/dexpace/morphic/actions/caches?sort=last_accessed_at&direction=asc').main's saves should shrink once they stop holding fuzz-instrumented builds, which is not measured. A night that restores nothing warns and searches from the seeds, which costs depth and nothing else. The workflow says this at the restore step, naming ci: reuse test results across checkouts and keep main's cache warm #1272, which changes whatmainsaves.Test plan
workflow_dispatchneeds the workflow on the default branch, so a throwaway branch ran the job on push at 3s a target, and was deleted with its comments and caches:CanonicalWordsreddensgo test ./ir/fuzz, and anirverifyplanted to report on every document reddensFuzzCompileandFuzzLowerSchemafrom their seeds alone. A compile planted to return neither a document nor a reason reddensFuzzCycleDetectoron all 79 seeds. A single-argument defect only a search reaches was found by the new binary in 6 of 6 runs (median 25s) and by the old in 4 of 6 within the 90s cap. A defect inFuzzHasCanonicalWords_AgreesWithCanonicalWordsneeding two correlated strings was not found in 60s, which that target's comment expects of a search, so its seeds carry those cases; the old binary was not run on that plant.fuzz.sh, each limit on both sides, on macOS's bash 3.2 and on Linux in the runs above: 16 targets pass and 17 refuse, a plant undertestdatais not counted wheremain'sgit grepcounts it, a stale quarantine refuses, every target quarantined refuses, a grammar that stops matching refuses, and a failinggo listexits non-zero.prunekeeps 1,000 entries and an entry of exactly 64 KiB, drops one a byte over, and drops a moved or deleted target.make fuzzin 0.2s, as onmain. Before the fix on this branch it carried on to the next target and was still running 45s later.mainit came from, the nine ref-shaped specs byte for byte../scripts/fuzz.sh chekexits 2 in under a second, where it took 240s to fail before, and durations such as5s,1m30sand0.5sand counts such as100xstill pass, on bash 3.2 and 5.3../scripts/fuzz.sh 1xsearched all 15 targets on the final tree.Test,Fuzz,BenchmarkandExampledeclarations go from 4,068 onmain(39afce9b) to 4,074, the additions beinginternal/ircases' andinternal/testspec' own tests.FuzzCycleDetectorruns 79 seeds and corpus entries against 44 onmain.irstays at 100% coverage without its fuzz seeds.actionlintongate.yml,make fmt,make vet,make lint-planted,make fuzz-check,golangci-lintv2.14.0 (0 issues),skills_lint, andgo testoverinternal,ir,compilersandcmd/morphic-census. That last run failed once with a package I did not capture, during a window when the disk filled, then passed in full on the same head; this PR's CI is the check on it.make coverageand the rest of the gate: left to this PR's CI, since the machine these ran on was too loaded for its timing tests.scheduleandworkflow_dispatchtriggers, and the concurrency group they select, have not run. They need the workflow on the default branch, and the trial runs above came in throughpush. After merging,gh workflow run gate.ymlexercises them without waiting for the night.Two open pull requests conflict with this one, each on lines both change, and whichever lands second keeps both sides:
suitejob beside theif:this adds, and addsraceto thegateand.PHONYlines this changes fromfuzztofuzz-check. A trial merge resolved that way runsracethenfuzz-checkinmake gate, passesactionlint, and leaves no prose saying the gate searches.CLAUDE.md.Related: #1315, #1316