Skip to content

ci: warm dataset cache directly and decouple Docker smoke test from test suites - #1275

Merged
ale18V merged 5 commits into
ci/selective-pr-checksfrom
ci/decouple-datasets-and-docker
Nov 4, 2026
Merged

ale18V merged 5 commits into
ci/selective-pr-checksfrom
ci/decouple-datasets-and-docker

Conversation

@ale18V

@ale18V ale18V commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

What & why

Currently in verify.yml:

  1. download-datasets restores datasets/ from actions/cache@v5, and then re-zips and re-uploads the entire folder to actions/upload-artifact@v7 on every single CI run so test jobs can pull it with actions/download-artifact.
  2. build waits on download-datasets (needs: [download-datasets, static-analysis]) even though _build.yml never downloads or uses datasets/.
  3. build-server-docker (~2m11s) is bundled inside _build.yml alongside build-topological (~1m06s), forcing all 14 test jobs in _test.yml to sit idle for an extra ~1 minute waiting for the Docker smoke test even though all-builds is already ready and no test job uses the Docker image.

Changes

  • Warm-only dataset cache (.github/actions/download-datasets/action.yml & pull-dataset-artifacts/action.yml):
    • Use lookup-only: true on actions/cache@v5 in download-datasets so cache hits complete in seconds without downloading or re-uploading a duplicate workflow artifact.
    • Restore datasets/ directly from actions/cache/restore@v5 (with fail-on-cache-miss: true) in pull-dataset-artifacts.
  • Remove unnecessary sequential dependencies (.github/workflows/verify.yml & _build.yml):
    • Move download-datasets from build.needs to test.needs.
    • Move build-server-docker out of _build.yml into verify.yml (needs: [static-analysis]) so it runs in parallel with build and test instead of blocking test startup.

@ale18V
ale18V requested a review from JulienVig October 29, 2026 12:49

@JulienVig JulienVig left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea to reduce unecessary dependencies! I think we can push this further for the Docker smoke test. However it looks like some dependencies are missing for it in ci-paths.yml .

Looking at the past CI runs it looks like the speedup is pretty unsignificant compared the timing of test/server. It doesn't cost anything to parallelize more but maybe we should focus our efforts on splitting test/server in several parallel jobs?

Comment thread .github/workflows/_test.yml Resolved
Comment thread .github/ci-paths.yml Resolved
Comment thread .github/workflows/verify.yml Resolved
@ale18V
ale18V force-pushed the ci/decouple-datasets-and-docker branch from 1a4e150 to 238d156 Compare November 3, 2026 13:03
@ale18V
ale18V requested a review from JulienVig November 3, 2026 13:48
@ale18V
ale18V added this pull request to stack #1278 November 3, 2026 15:03
@ale18V
ale18V merged commit 6d111e5 into main Nov 4, 2026
22 checks passed
@ale18V
ale18V deleted the ci/decouple-datasets-and-docker branch November 4, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants