ci: move the fuzz search from pull requests to a nightly job by fuad-daoud · Pull Request #1382 · dexpace/morphic

fuad-daoud · 2026-11-02T22:30:29Z

Summary

Every pull request spends a median 231s on a fuzz search (the 79 runs in #1273) that restarts from the seeds each time, so the search never adds up. This moves it to a nightly job that keeps its corpus.

Stacked on #1373 (#1272), which is stacked on #1311: all three change the same gate.yml and Makefile lines. The diff here shows only #1273's changes; once those two merge, this retargets to main.

This carries the nightly search from #1301 (Omar's), as its review on that pull request proposed.

The search leaves pull requests

make fuzz-check replaces the search in the gate. It holds the targets to the cap, the quarantine to real targets and the list to at least one searchable target, and searches nothing. Every target's seeds still run in go test ./.... Fuzzing leaves make gate.
A third job, fuzz search, runs on a schedule or by hand and never beside the other two. It has its own concurrency group, since a scheduled run carries main's sha and would otherwise share gate-<sha> with main's push run. At 1m a target it stays inside the 30-minute job timeout at the target cap: 16 × 1m of search, plus a 6m26 cold build measured on 4 pinned cores (./scripts/fuzz.sh 1x on a fresh GOCACHE), is about 22m26, against the search step's 25 and the job's 30. Minimizing adds nothing to that: -fuzztime's deadline cancels it too (go1.27 internal/fuzz/fuzz.go: the workers' context is a child of the -fuzztime timeout), so a found crash cannot lengthen a target's search.
The corpus carries over. The job restores and saves $GOCACHE/fuzz alone, under a fuzz-corpus- key no other job restores. Go never trims that directory, so make fuzz-prune drops targets that no longer exist and entries over 64 KiB, and keeps 3,000 entries a target (the cap sits above where a target still builds on its corpus, since FuzzCycleDetector reached 908 entries in five simulated nights with its coverage still rising, and below where re-running every entry, which happens inside the search budget, would leave little time to search). With 3,001 entries for one target on a fresh GOCACHE, ./scripts/fuzz.sh prune prints FuzzDecimal_OrdersAsRationalsDo is past 3000 entries; dropped 1 and corpus: 12000 kB in 3000 entries. A deliberate exception to "the cache never grows" (ci: run the gate as two jobs and drop the Windows and wasm ports #1025, ci: carry main's Go build cache into every run, update actions #755).
A failed search reports itself. It uploads what it found and comments on ci: report each failed nightly fuzz search here #1296, which stays open for this. The comment says which inputs were written, or that none were, so a broken build is not mistaken for a finding. The job alone holds issues: write, and it runs only main's code or a writer's dispatch, never a pull request's. The checkout leaves no credential in .git/config and the search step's environment holds no token, though actions/checkout and setup-go, each as its token input's default, and the reporting step are handed one.
fuzz.sh reads its targets from the packages ./... names, so a plant under testdata no longer spends the budget or trips the cap. It also searches every target after a failure, stops at an interrupt, and refuses an argument that is neither check, prune, nor a duration or count above zero as go test reads -fuzztime (a sign allowed), before anything builds.
The lint step sets GOFLAGS: -trimpath, as the Makefile does for every other step, so it reuses what make vet compiled (ci: move the fuzz search from pull requests to a nightly run #1273's last item). One run each: the step took 219s on the base's run 41489972604 and 53s on this head's run 41537851553.
The lint, fuzz, bench check is now lint, bench. No ruleset requires either name: gh api repos/dexpace/morphic/rulesets/19179519 --jq '[.rules[].type]' gives deletion, non_fast_forward, pull_request and code_quality, and main's branch protection lists no required checks.
main no longer drops the fuzz corpus. The step that did is gone, and so are the four comments it made false: only the fuzz search job runs -fuzz, and it saves its corpus under its own key, so nothing main saves holds one. On a fresh GOCACHE, fuzz.sh check, make bench-smoke and go test ./... -run '^Fuzz' -count=1 each leave no $GOCACHE/fuzz, and a 3s -fuzz run, the control, does. Two ir targets' docs that said the gate searches now say scripts/fuzz.sh does.

-fuzzminimizetime (left at 10s here) is the pull request stacked on this one, for #1290, and the five ir and three e2e target moves are #1381.

Decisions worth reading

A deadline with no finding gets no handling of its own. The pinned Go suppresses it (go.dev/issue/75804), so a red night is a finding or a failure worth reading; ci: fuzz step reddens on the coordinator's own deadline with no finding #466 can be judged on that evidence separately.
What this gives up: a bug only a search would find now reaches main and waits for the night, where a 10s search on the pull request might have caught it. ci: cut a pull request's gate from a 9-minute median to about 3.5 #1268 weighs that trade.
The corpus can be evicted, and on this history often will be. It shares the repository's 10 GiB cache limit with what main saves, and eviction takes what was used longest ago. Today the caches already total 11.5 GB (gh api repos/dexpace/morphic/actions/cache/usage), and one push to main saves about 1.1 GiB, so roughly nine pushes after a night's save evict its corpus. Of the last 21 days, counted from 02:17 UTC, 7 had nine or more pushes to main. So on that history about one night in three would restore nothing and search from the seeds: it warns and carries on, but those nights lose the depth this change is for. main's saves may shrink once they hold no fuzz-instrumented builds, which is not measured. ci: measure whether the nightly fuzz corpus survives the cache limit #1383 tracks measuring it after merge and keeping the corpus out of eviction's reach if it holds.

Authorship

Omar's commits are carried with him as author: 55f19d31, 3cacbf48, b124b961, 76556497, 3860708c, 5849a029, 4675381e. 55f19d31 is the first part of his 22cd99d0, split in two so its -fuzzminimizetime=1s and 5s-by-hand part could be its own pull request, the one stacked on this. The split was built on a line at 22cd99d0's parent, where the two parts applied in order give 22cd99d0's tree exactly. git range-diff against his originals shows = for e8b51544, 4187fe0f, dc456509 and 5a11c04e; c6e0265d and 967c30d8 differ in one context line each, the minimize_time and FUZZTIME defaults, which read 10s here; the split part differs where its hunks reapply onto #1272's header. Our commits on top: a budget grammar fix (ace28bc3, ae17963c), the checkout-token comment (bc28ebb2), this change's two.

Before and after

flowchart LR
  subgraph before["Before"]
    direction TB
    b1["every pull request"] --> b2["make gate searches 15 targets × 10s<br/>median 231s of each run"]
    b2 --> b3["each search restarts from the seeds<br/>nothing carries to the next"]
  end
  subgraph after["After"]
    direction TB
    a1["every pull request"] --> a2["make fuzz-check: caps and quarantine checked,<br/>seeds run in go test, no search"]
    a3["nightly or by hand"] --> a4["fuzz search job: 1m a target"]
    a4 --> a5["corpus restored and saved under its own key,<br/>pruned to 3,000 entries a target"]
    a4 -. "on failure" .-> a6["findings uploaded, comment on #1296"]
  end
  before ~~~ after

What it brings: pull requests stop paying about four minutes for a search that never builds on itself, and the search runs nightly for a minute a target on a corpus that grows from night to night.

Test plan

The step was a no-op: a fresh GOCACHE leaves no fuzz directory after scripts/fuzz.sh check, make bench-smoke or go test ./... -run '^Fuzz' -count=1, and keeps one after a 3s -fuzz run.
./scripts/fuzz.sh 1x searches all 15 targets and exits 0, in 6m26 on a cold cache.
git grep -nE 'drop-fuzz|drop the fuzz corpus|only with -fuzz and -bench|drops \$GOCACHE/fuzz|dropped before the save|no cache of its own yet' -- .github and git grep -nE "gate's (per-target )?search|bounded to seconds" print nothing; actionlint is clean; go test ./internal/archtest ./ir/... ./internal/harness passes.
The job itself ran on real Actions, before the split, on ci: move the fuzz search from pull requests to a nightly job #1301's trial branch through push, at 3s a target: run 41354481475, a planted crash, searched all 15 targets past it, uploaded the artifact, commented with the reproducer and saved the corpus after the failure; run 41354927259 restored that corpus and was green; run 41355384470, a broken build, commented that no reproducer was written. Those runs predate the restore warning and the token comment. In 41354481475 the search step took 244s for 15 × 3s, so CI's cold build was about 200s, under the 6m26 measured locally.
Not run: the schedule and workflow_dispatch triggers, and the concurrency group they select, have not run. They need the workflow on the default branch. After merging, gh workflow run gate.yml exercises them without waiting for the night, and the job's comment on ci: report each failed nightly fuzz search here #1296 with them.
make gate: this pull request's CI runs it on this head.

Crossings with the other two pull requests

README's "Fuzzing" bullet: this one rewrites the text and keeps ./compilers/openapi/e2e; test: move fuzz targets to packages that link no test tooling #1381 changes only the path, to ./compilers/openapi/e2e/fuzz. Whichever lands second keeps this text with that path.
The ir fuzz targets' doc lines: edited here in ir/bigval_property_test.go and ir/naming_property_test.go; test: move fuzz targets to packages that link no test tooling #1381 writes the same lines in ir/fuzz/bigval_test.go and ir/fuzz/naming_test.go, so the second merge resolves to the same text.
The corpora of test: move fuzz targets to packages that link no test tooling #1381's eight moved targets: fuzz.sh prune drops a target that is gone from its old import path, so the first night after test: move fuzz targets to packages that link no test tooling #1381 lands starts those eight from their seeds. Landing test: move fuzz targets to packages that link no test tooling #1381 first, before this saves a corpus, avoids that.

Closes #1273

Related: #1272, #1311, #1296, #466, #1290, #1301

Out of scope: nothing on a pull request runs fuzz.sh prune or the search, so a defect in either first shows on a night. #1384 tracks a fuzz-check self-test that prunes a fixture cache in $TMPDIR and asserts what it prints.

Every pull request spent a median 231s on a search that restarted from the seeds each time. Pull requests now run make fuzz-check, which holds the targets to their cap and the quarantine to real targets without searching, and every target's seeds still run in the suite. Fuzzing leaves make gate. The search runs on a schedule or by hand, in a job of its own, with a corpus it restores and saves between nights under a key nothing else reads. Go never trims that directory, so make fuzz-prune drops targets that are gone, entries over 64 KiB, and anything past 1,000 entries a target. A failed search uploads its findings and comments on one standing issue. That job alone may write to issues, only the reporting step holds the token, and the checkout leaves no credential. fuzz.sh reads its targets from the packages ./... names, so a plant under testdata no longer spends the budget or trips the cap, and it searches every target after a failure, so one finding cannot cost the rest of the night. The lint step sets the -trimpath the Makefile exports for every other step, so it can reuse what make vet compiled. Split from 22cd99d, which also shortened minimizing to 1s and the budget of a search run by hand to 5s: those are a change of their own.

go test handles an interrupt and exits, which the search reads as a failed target and carries on from, so Ctrl-C stopped one target of fifteen. Before the search learned to carry on past a failure, set -e stopped it at once. Trap the interrupt so the search exits 130 once go test has gone.

The corpus shares the repository's 10 GiB cache limit with what main saves, at about 1.1 GiB a push, and eviction takes what was used longest ago. Enough pushes in a day lose it, and the search then starts from the seeds, which is what it did before it kept a corpus. That costs depth and nothing else, so it should not fail the run, but it should not pass unnoticed either.

prune decides between keeping a target and rm -rf with printf piped to grep -q. Under pipefail, grep leaving early kills printf, the pipeline reports failure, and a live target reads as gone. The list is small enough today that printf finishes first, but a larger one missed its first line 5 times in 5. A here-string has no writer to kill.

1,000 entries a target was a guess, and five simulated nights refuted it: FuzzCycleDetector reached 908 with its coverage still rising, from 42,394 bits to 48,137, so the cap would have started evicting at random around night six. Its gains fell by about a fifth a night (264, 220, 168, 145, 111), which points to roughly 1,400 entries at that search rate. A search re-runs every entry inside its budget, 9s at 900 entries under load, so the cap cannot rise freely either; 3,000 leaves twice that margin. A target past it now says so in the log.

fuzz.sh takes check, prune or a -fuzztime budget as its one argument. A mistyped mode was handed to go test as a budget, so every target built before rejecting it: chek took 240s to fail. It is now refused at once, with exit 2.

The comment said only the reporting step held the token. setup-go is handed it too, as its token input's default, so that was not so. The job's safety rests on which code it runs: main's, or a writer's dispatch, never a pull request's.

The comment above the fuzz job listed setup-go and the reporting step as the steps handed a token. It left out checkout, whose token input also defaults to github.token, so the list was incomplete. It now names all three, and says that checkout and setup-go get the token as an input's default and the reporting step as GH_TOKEN. The job's safety argument is unchanged: checkout runs before any of the repository's code, and persist-credentials: false keeps the token out of .git/config.

The budget expression refused durations that -fuzztime accepts: a fraction with no integer part (.5s), one with no fraction digits (1.s), and the microsecond unit spelled with either mu sign (5µs, 5μs). The comment above it says the script reads the budget as go test does, so a budget go test takes was refused before any target ran. The expression now accepts a count (100x) or one or more durations, each a number with an optional fraction (5, 5., 5.5 or .5) and a unit of ns, us, µs, μs, ms, s, m or h. It refuses a bare number, a sign, a lone point and a count with a fraction (1.5x). 0 stays refused although go test reads it, as the expression adopted in review does.

go test refuses -fuzztime=0 and every other zero budget (0s, 0x, 0.0s). The previous expression accepted all of them except 0, so each target built only to be refused. The script now refuses every zero budget, as go test does, and no longer claims that go test reads 0: it does not. A leading plus, which both time.ParseDuration and strconv.ParseInt accept, is now accepted too. A value out of range, too large or below a nanosecond, is left to go test to refuse, as the comment now says: telling it apart takes arithmetic, not a pattern.

Only the fuzz search job runs -fuzz, and it saves its corpus under its own key, so nothing the checks and suite jobs save on main holds one. The step that removed $GOCACHE/fuzz before the checks job's save therefore removes nothing: a fresh cache stays without the directory after fuzz-check, bench-smoke and every target's seeds run. Three comments in the workflow's header said otherwise and now say what holds: that the checks job runs go test only with -bench and the search job only with -fuzz, neither of which go help test lists as cacheable; that a run on main only saves; and that the corpus never enters those saves, being bounded by make fuzz-prune in the job that writes it. The comment that the corpus had no cache of its own yet goes with the step.

Two fuzz targets' docs said the gate's search is bounded to seconds. The gate searches nothing now: an ordinary go test runs the seeds, and scripts/fuzz.sh runs the search.

The comment said #1272 changes what main's pushes save, as if it were still to come. This pull request is stacked on #1373, which lands #1272, so on the tree that merges the change is already in effect, and the workflow's header already says what main saves.

OmarAlJarrah and others added 12 commits November 2, 2026 22:12

ci: refuse a fuzz.sh argument that is no mode or budget

5849a02

fuzz.sh takes check, prune or a -fuzztime budget as its one argument. A mistyped mode was handed to go test as a budget, so every target built before rejecting it: chek took 240s to fail. It is now refused at once, with exit 2.

test(ir): say what runs the fuzz search without naming the gate

1bbe092

Two fuzz targets' docs said the gate's search is bounded to seconds. The gate searches nothing now: an ordinary go test runs the seeds, and scripts/fuzz.sh runs the search.

fuad-daoud mentioned this pull request Nov 2, 2026

ci: move the fuzz search from pull requests to a nightly run #1273

Open

This was referenced Nov 2, 2026

ci: measure whether the nightly fuzz corpus survives the cache limit #1383

Open

test: run fuzz.sh prune on a fixture cache in make fuzz-check #1384

Open

fuad-daoud marked this pull request as ready for review November 2, 2026 23:03

fuad-daoud requested a review from OmarAlJarrah November 2, 2026 23:03

This was referenced Nov 2, 2026

ci: minimize a fuzz input for 1s, not 10s #1385

Open

ci: cut a pull request's gate from a 9-minute median to about 3.5 #1268

Open

ci: run the race step in the checks job, beside lint #1386

Open

Merge branch 'ci/reuse-test-results' into ci/nightly-fuzz-search

ddb64ab

fuad-daoud mentioned this pull request Nov 3, 2026

ci: run the race step in the checks job #1387

Open

Navigation Menu

Repository navigation

ci: move the fuzz search from pull requests to a nightly job - #1382

ci: move the fuzz search from pull requests to a nightly job#1382
fuad-daoud wants to merge 14 commits into
ci/reuse-test-resultsfrom
ci/nightly-fuzz-search

fuad-daoud commented Nov 2, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

fuad-daoud commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

The search leaves pull requests

Decisions worth reading

Authorship

Before and after

Test plan

Crossings with the other two pull requests

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fuad-daoud commented Nov 2, 2026 •

edited

Loading