Repository navigation
[CI] Cache torch.compile artifacts for the pytorch unit tests - #2170
Merged
felixdittrich92 merged 1 commit intoOct 16, 2026
Merged
Conversation
The pytorch test job compiles 23 models with torch.compile, and every run starts from an empty TorchInductor cache, so each graph goes through the full Inductor lowering, C++ codegen and gcc build again. On CPU this is the largest block of the job: the tablecenternet compile test alone takes over 10 minutes cold. setup-env gets an opt-in cache-compile input that points TORCHINDUCTOR_CACHE_DIR at ~/.cache/torchinductor and persists it with actions/cache. Only pytest-torch enables it. The key carries the runner image, Python version, torch version and the vector ISA Inductor picks for code generation, plus a hash of doctr/**/*.py. A prefix restore-key lets a run reuse the entries of a previous commit: Inductor looks entries up by the traced graph, torch build and its config, so graphs a change does not touch are reused and changed ones are compiled again. Measured in a Linux container (python:3.11, 4 CPUs, torch 2.14.1+cpu), with the CI command (pytest tests/pytorch/ -rs --cov): - the 23 compile tests: 2315s with an empty cache, 535s with a cache restored from a previous run (-77%); the cache is 96 MB - tablecenternet compile test: 665s -> 52s - full suite with the cache restored into a fresh container: same 452 test ids, same outcomes (447 passed, 5 skipped) and same coverage totals (8711/10741 lines) as the cold baseline run
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2170 +/- ##
=========================================
+ Coverage 0 97.61% +97.61%
=========================================
Files 0 170 +170
Lines 0 10740 +10740
=========================================
+ Hits 0 10484 +10484
- Misses 0 256 +256
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
felixdittrich92
approved these changes
Oct 16, 2026
felixdittrich92
left a comment
felixdittrich92
left a comment
Collaborator
There was a problem hiding this comment.
Thanks @quotentiroler 👍
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The pytorch test job compiles 23 models with torch.compile and starts every run with an empty TorchInductor cache. So each graph goes through Inductor lowering, C++ codegen and a gcc build again. On CPU this is the largest part of the job:
test_torch_compiled_models[tablecenternet]alone takes over 10 minutes cold.This adds an opt-in
cache-compileinput to the setup-env action. It pointsTORCHINDUCTOR_CACHE_DIRat~/.cache/torchinductorand persists it with actions/cache. Onlypytest-torchenables it.The key includes the runner image, the Python and torch versions, the vector ISA Inductor generates code for, and a hash of
doctr/**/*.py. A prefix restore-key lets a commit reuse the previous entries. Inductor looks entries up by the traced graph, the torch build and its config, so models a change does not touch are reused and changed ones are compiled again.Measured in a Linux container (python:3.11, 4 CPUs, torch 2.14.1+cpu) with the CI command:
The tests and assertions are unchanged. The first run on main fills the cache.
Prepared with Claude Code; I reviewed and tested the change.