ci: reuse test results across checkouts, keep main's cache warm by fuad-daoud · Pull Request #1373 · dexpace/morphic

fuad-daoud · 2026-11-02T14:27:44Z

Summary

CI reran tests whose inputs had not changed, and main built every run from an empty cache. Go reuses a cached test result only when every file the test opened inside the module has the same size, mode and modification time, and a fresh checkout gives every file a new time. This makes a checkout reuse results, and keeps main's cache warm without ever replaying a stale pass on main.

Date files by content. make filetimes (new cmd/morphic-filetimes) sets each tracked file's time from its git blob hash, every directory's to a fixed time, and .git's last. It refuses a dirty worktree, since the hash comes from the index. CI's suite job runs it after build, because go build's VCS stamping runs git status, which re-dates .git, and three of the slowest packages list the module root.

Carry main's cache forward. A run on main restores the previous main run's cache, then go clean -testcache voids every result in it, then the whole suite runs and the cache is saved. It is not -count=1, which stores no results and would leave pull requests nothing to reuse. The key carries hashFiles('go.sum'), so a dependency change starts afresh, and $GOCACHE/fuzz is dropped before the save. A pull request restores that save and never saves.

Tests that read through a child process now open what their verdict rests on, so dating cannot replay a stale pass:

internal/lintrules and internal/archtest read the parser library's sources from the module cache, outside the module, where Go never rechecks. Each now opens go.mod and go.sum, and irtest.RequireOpensModuleFiles holds that through the test log.
ir/irtest's line-ending tests rest on lists git produces, so a commit that only adds a file was invisible to the cache. TestTrackedFiles_HoldNoCarriageReturn now lists each directory above a tracked file.

morphic-verifying-changes/references/toolchain.md#test-opens-its-inputs states the rule, with a probe that commits an added file, dates the tree, and reruns.

Closes #1272.

Before and after

flowchart LR
  subgraph before["Before"]
    direction TB
    b1["checkout"] --> b2["every file gets a new time"]
    b2 --> b3["every test that opens a fixture reruns<br/>31 of 61 packages for a one-file change"]
    b4["main's run"] --> b5["starts from an empty cache<br/>rebuilds everything"]
  end
  subgraph after["After"]
    direction TB
    a1["checkout, build"] --> a2["make filetimes<br/>time from each file's content"]
    a2 --> a3["unchanged tests reuse their results<br/>56 of 61 packages cached"]
    a4["main's run"] --> a5["restore main's cache<br/>go clean -testcache, run everything, save"]
    a5 -. "results to reuse" .-> a3
  end
  before ~~~ after

What it brings: a pull request reruns only the tests its change can reach, and main keeps its builds between runs while still rerunning every test. Measured at 5552c6e4 in fork runs 41323114637 and 41326728885, before #1311 split the suite, which run to run varies by about ±30%: a one-file change's suite fell from 284s to 129s and from 386s to 83s, with 56 of 61 packages cached instead of 30, and main's suite from 500s to 391s. The diagram's package counts are from those runs.

This pull request's own CI ran cold, so its timings are no measure of this change: actions/cache reported Cache not found for both jobs (run 41489972604, on this head), since the key now carries go.sum's hash and no run on main has saved under it yet. The first push to main after merge saves one, and the pull requests after it restore it.

Stacked on #1311

This is based on #1311's branch, since both change the suite job and the Makefile's gate. The diff here shows only #1272's changes. In suite, the race run comes after date files by content and clear restored test results, since otherwise a run on main could replay a race-free result the clear never voided. Once #1311 merges, this retargets to main.

Test plan

CI run 41489972604 passes on this head, de79b4a9, which holds ci: run coverage in set mode and -race on concurrent packages #1311's branch and main at 3188471d. make gate passed on the earlier stacked tree, with main at 6996ce45, pinned to four cores with its own GOCACHE; the merges since were clean, and archtest, ir/irtest, cmd/morphic-filetimes and internal/lintrules pass on this head. On this head, ci: run coverage in set mode and -race on concurrent packages #1311's TestRaceList_* passes with nothing to add: cmd/morphic-filetimes takes no lock and starts no goroutine. actionlint is clean. In suite, the steps run build, date files, clear restored results (on main only), coverage, race, save. Only checks, which fuzzes, drops $GOCACHE/fuzz before its save.
Reuse after a fresh checkout, in three of three cycles: prime, delete the clone, clone again to the same path, date, test. ir/irtest, internal/lintrules and internal/keydoc all show (cached); an undated control reruns them.
Rerun after a real change, three classes: a committed // want deletion in lintrules' plants fails without -count=1; a committed unruled package fails archtest; a go.sum touch reruns lintrules.
An added file reruns ir/irtest: a committed CRLF blob, and a nested .gitattributes with eol=crlf, each FAIL without -count=1. Before the directory listing, both gave ok (cached).
main's clear leaves no stale pass: with the CRLF commit dated and no clear, a primed cache gives ok (cached); after go clean -testcache, FAIL. A result saved after the clear is reused next time.
The directory listing is held: TestTrackedFiles_ListsEveryDirectoryAboveAFile runs the line-ending test in a child and requires an open for every directory above a tracked file, through irtest.RequireOpens, which RequireOpensModuleFiles now calls with go.mod and go.sum. Listing only the root, or only each file's parent, fails it. Requiring go.mod alone fails TestRequireOpensModuleFiles_RefusesThroughARealChild.
The module-file opens are held: removing ReadModuleFiles from readDeclarers, libraryMembers or moduleProgram fails its …_OpensTheModuleFiles test.
cmd/morphic-filetimes: five of six mutants killed (nanoseconds dropped, other index tags accepted, a dirty path allowed, untracked files ignored, the directory time moved). The sixth, directories dated before files, is equivalent on Linux, where dating a file leaves its directory's time alone.
Cost of the new pins: each RequireOpensModuleFiles reruns a named test in a child: 0.34s in lintrules, 3.44s and 5.31s in archtest (warm, without -race).

After merge (the push path runs only on main):

restore hits the previous main run's key by prefix, with the same go.sum hash, and clear restored test results runs before any test.
suite's test steps show no (cached) line.
save caches writes …-<sha>. Its size stays flat across several pushes, read with gh api repos/dexpace/morphic/actions/caches --jq '.actions_caches[] | [.key,.size_in_bytes] | @tsv'.
The next pull request restores that key and shows (cached) for packages it does not reach.

Out of scope: the scripts/vet-rules.sh fix for a vet finding replayed under another checkout's path is #1305's own change. The nightly fuzz corpus is #1273's: once it lands, drop the fuzz corpus deletes nothing and goes.

Related: #1268, #1311, #1273, #1305, #1025, #755

The rules' tests read the parser library's sources through go list and go/packages. They sit in the module cache, and go test rechecks only the files inside the module that a test opens, so a dependency bump replayed a stale pass whenever no file inside the module had changed. Add irtest.ReadModuleFiles, which opens the module's go.mod and go.sum in the test's own process, and call it from readDeclarers. readDeclarers now returns the paths it read, and a test holds them to both files. The same mechanism sat in internal/archtest, where the resolution tests derive the library's API from its sources and the reachability check parses every dependency of the module. go.mod was rechecked there only because other tests stat it, and go.sum never was. Both now open the two files themselves.

irtest.RequireOpensModuleFiles re-runs a named test with -test.testlogfile and requires its log to open go.mod and go.sum. The test cache keys a result on that log, not on a value a helper returns, so a test that only compared the returned paths passed with the read gone. lintrules and archtest each hold their module-file opens through it: readDeclarers, libraryMembers and moduleProgram each have a test that reads the log of a test reaching them. ReadModuleFiles and readDeclarers no longer return paths, as nothing read them once the log held the opens. readDeclarers' comment now names the order it reads in.

The child RequireOpensModuleFiles starts now carries the parent's remaining time as its -test.timeout, so a parent killed by its own timeout no longer leaves the child running unbounded: the test binary ran on after its parent died, because the context that kills it lives in the parent. A parent with no limit gives the child none. The helper refuses to start a child when the parent has no time left, as a timeout of zero or less turns the child's timeout off.

RequireOpensModuleFiles reads its own deadline and hands the child what it has left as -test.timeout, but the tests of requireOpens and childArgs inject both values themselves, so a wrong value in the wrapper stayed green at full coverage. Add a pair that goes through a real child. The child opens the module's files only if it carries the marker, a parent deadline in its environment, a deadline of its own, and that deadline falls no later than the parent's plus a minute for startup. The parent sets its deadline in the environment and requires the child's log to show the opens, so a child given no timeout, or a longer one, fails the suite. The parent skips when the run has no deadline.

The tests of requireOpens inject the caller's name, its marker and the log's read, so a wrong value built by RequireOpensModuleFiles stayed green at full coverage: the marker that stops a child starting another, the name that stops a test running itself, and the read of the child's log each had a mutant no test saw. Run the call in a process of its own for a caller naming itself, a caller inside a marked child, and a child whose log opens nothing, and require each to fail that process for the reason named.

A script would go untested under the gate, whose coverage rule and architecture test reach Go commands alone. A Go command also needs no GNU or BSD touch and stat flags, and runs on the toolchain the Makefile pins.

go vet replays a cached finding under the path of the checkout that first reported it, so a checkout of the same content at another path failed the check on the plant's absolute path. Match the file by its path below the module root and require exactly one finding.

A git hook exports GIT_DIR and GIT_INDEX_FILE, so the tests' git, run from one, wrote fixture commits into the hook's repository. TestMain now unsets every inherited GIT_ variable before any test runs, which covers each place the tests reach git: the fixture repository, the command's own git, and the runs through the current directory.

mapfile is bash 4, and the bash a stock macOS ships is 3.2, so the script stopped before checking anything there. One case over the finding list keeps the exactly-one-finding guarantee and runs on both.

Tests that list the module root hash every entry's time, .git included. A checkout and any git status, go build's VCS stamping among them, give .git a new time. So the command now dates .git last, and in suite the step runs after build.

A run on main now restores the previous run's cache of the same job, toolchain and go.sum, clears its test results with go clean -testcache, runs every test, drops $GOCACHE/fuzz and saves. go.sum joins the cache key, so a dependency change starts afresh and bounds the module cache. The results are cleared rather than skipped with -count=1, which stores none and would leave pull requests nothing to reuse.

The rule told every test that reads through a child process to open go.mod and go.sum. A test whose input is an in-module file that `go list` reads stays cached when that file changes, so the rule now has the test open the child's inputs, and names the helper only for a dependency.

…/c1272b

go test keys a cached result on what the process opens. Once every tracked file and directory has the same time on every checkout, a commit that adds a file, or a .gitattributes, opens nothing new and leaves the result of TestTrackedFiles_HoldNoCarriageReturn and TestGitAttributes_ResolveToLF cached. Listing each directory above a tracked file changes a listing the log holds when git lists a file for the first time. The test's doc also said it can never redden on the Linux runner. CI's checkout normalizes text files, but a CRLF blob committed with its filters bypassed, or a binary fixture holding 0x0D, reddens it.

go vet replays a finding under the first checkout's path. That happens with no dating at all, so dating tracked files neither needs nor exposes the fix, and it lands as its own change. This restores scripts/vet-rules.sh to main.

The checks job runs go test only with -fuzz and -bench, which go help test does not list as cacheable, so dating files and clearing restored test results there acted on nothing. Both run in the suite job alone, defined where they are used. Dropping the fuzz corpus stays in both jobs, as fuzz runs in checks.

Go rechecks a cached test result against the files inside the module root that the test opens, as irtest.RepoRoot's neighbour ReadModuleFiles already says. The workflow header, the dating step's comment, the filetimes target's comment and a new test's comment said every file or each file a test opens.

libraryMembers' comment said a dependency bump reruns the tests only through the go.mod and go.sum opens. Archtest's walk of the module root lists the root, and go test hashes each entry of a listed directory, go.sum included, so a bump already reruns the package. The opens tie these tests to a bump themselves, rather than through that walk.

A touch of the file the verdict rests on cannot find a child's list: the package already opens every file on it, so the touch reruns it. Add the probe that finds it, a file the child lists for the first time, and record that the child-process sites were swept and the one hole found is closed.

The comment said a checkout dates every file alike. A checkout gives every file a new time, and make filetimes gives each file the same time on every checkout, which is what makes an added file invisible to the test cache unless its directory is listed.

The previous commit wrote the target as `make filetimes`, which misspell reads as a typo of "lifetimes". The linter's exclusion for that word is limited to the command's own package and the architecture rule, so the comment names the target by what it does instead.

gate.yml: the suite job keeps its id and the separate coverage and race steps of the split, and the order of the dating change: build, date files by content, clear restored test results, coverage, race, drop the fuzz corpus, save caches. The race step is a test run, so it follows the clear. Nothing runs git between the dating step and the first test step. The clear's comment now says it precedes both test steps. Makefile: both targets stay, `race` and `filetimes`, in the header list and in .PHONY. `gate` lists `race` and not `filetimes`, which is cache plumbing that CI runs before its tests and not a check.

…reuse-test-results

RequireOpens takes the paths a child must open, and RequireOpensModuleFiles calls it with go.mod and go.sum. TestTrackedFiles_ListsEveryDirectoryAboveAFile runs the line-ending test in a child and requires an open line for the root and every directory above a tracked file, so a listing that stops short fails instead of letting a cached result stand.

The suite job runs no -fuzz, and only go test -fuzz writes $GOCACHE/fuzz, so its drop step removed nothing. The checks job keeps the step, and the header now says that job alone drops the corpus.

TestRequireOpensModuleFiles_RefusesThroughARealChild is the one test that runs the path list RequireOpensModuleFiles builds. Its want for a child that opens nothing now names both go.mod and go.sum under the module root, so a list that drops either file fails it.

The header paragraph's last lines now wrap at the width the rest of the file's comments keep. Only comments change.

…reuse-test-results

The comment said #1272 changes what main's pushes save, as if it were still to come. This pull request is stacked on #1373, which lands #1272, so on the tree that merges the change is already in effect, and the workflow's header already says what main saves.

fuad-daoud added 28 commits November 1, 2026 21:52

build: add a command that dates tracked files by their content

2fc93ff

A script would go untested under the gate, whose coverage rule and architecture test reach Go commands alone. A Go command also needs no GNU or BSD touch and stat flags, and runs on the toolchain the Makefile pins.

fix(scripts): read vet-rules findings without a bash 4 builtin

1b6ae30

mapfile is bash 4, and the bash a stock macOS ships is 3.2, so the script stopped before checking anything there. One case over the finding list keeps the exactly-one-finding guarantee and runs on both.

ci: date files by content so checkouts reuse test results

0590c18

docs(cmd/morphic-filetimes): keep the package comment under the word cap

e55ecaa

chore(skills): make a test open the inputs its result rests on

c4646ee

Merge branch 'main' of https://github.com/dexpace/morphic into relevo…

0c5d36c

…/c1272b

chore(skills): reflow the test-opens-its-inputs rule's sentences

f0bd924

Merge branch 'main' into relevo/c1272d

4e3966d

Merge remote-tracking branch 'origin/ci/race-coverage-split' into ci/…

20edfc5

…reuse-test-results

fuad-daoud mentioned this pull request Nov 2, 2026

ci: reuse test results across checkouts and keep main's cache warm #1272

Open

fuad-daoud marked this pull request as draft November 2, 2026 15:07

fuad-daoud mentioned this pull request Nov 2, 2026

skills: no review probe simulates the runs a CI or test-cache change claims #1374

Open

fuad-daoud added 7 commits November 2, 2026 18:10

chore(skills): hold a child's directory listing through the log

04a74b2

ci: drop the fuzz corpus only in the job that fuzzes

583ea3b

The suite job runs no -fuzz, and only go test -fuzz writes $GOCACHE/fuzz, so its drop step removed nothing. The checks job keeps the step, and the header now says that job alone drops the corpus.

ci: rewrap the gate workflow's cache header to its width

9ca56d8

The header paragraph's last lines now wrap at the width the rest of the file's comments keep. Only comments change.

Merge remote-tracking branch 'origin/ci/race-coverage-split' into ci/…

9d0488a

…reuse-test-results

Merge remote-tracking branch 'origin/ci/race-coverage-split' into ci/…

de79b4a

…reuse-test-results

fuad-daoud marked this pull request as ready for review November 2, 2026 17:33

fuad-daoud requested a review from OmarAlJarrah November 2, 2026 17:33

This was referenced Nov 2, 2026

ci: minimize a fuzz input for 1s, not 10s #1385

Open

ci: cut a pull request's gate from a 9-minute median to about 3.5 #1268

Open

Navigation Menu

Repository navigation

ci: reuse test results across checkouts, keep main's cache warm - #1373

ci: reuse test results across checkouts, keep main's cache warm#1373
fuad-daoud wants to merge 35 commits into
ci/race-coverage-splitfrom
ci/reuse-test-results

fuad-daoud commented Nov 2, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

fuad-daoud commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Before and after

Stacked on #1311

Test plan

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

fuad-daoud commented Nov 2, 2026 •

edited

Loading