Skip to content

Give the UK qualify job 330 minutes - #660

Merged
MaxGhenis merged 2 commits into
mainfrom
fix/uk-qualify-timeout
Oct 18, 2026
Merged

MaxGhenis merged 2 commits into
mainfrom
fix/uk-qualify-timeout

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #659

The UK qualify job hits its 120-minute timeout on every attempt ("The job has exceeded the maximum execution time of 2h0m0s", run 39145419501). The catch-up then re-stages the attempt, so no candidate has finished its gates since 17 October around 14:30Z; 2.135.6 has been staged seven times.

Cause: before #651, every wrapper.household contract row errored at once (policyengine.uk was None), and the gates took 80–89 minutes. #651 made those rows run the household calculator, which takes the gates past two hours. Attempts with #651 but not #627 timed out too, so contract 2.0.0 isn't the cause. Projected total is about 135–145 minutes, per the independent review's log analysis. The cancelled log shows active computation up to the cut-off, not a hang.

Change: timeout-minutes: 330 on the qualify job, with a comment saying why. That's under the 360-minute ceiling for GitHub-hosted jobs. The job's concurrency group (uk-candidate-<pair>, cancel-in-progress: false) already stops later runs from cancelling it, and the replay job keeps its 355. No code or test changes. The workflow tests don't pin this value, and the YAML parses with qualify at 330.

Follow-ups (not here):

Independent review (Opus, in-session): VERDICT: AGREE. It confirmed the timeout annotation on 4 jobs, that all 15 qualify jobs since #651 were cancelled at about 2h00m20s, and that the ledger durations bear this out. The YAML is valid, the workflow tests pass 24/24, and concurrency and replay are unaffected.

🤖 Generated with Claude Code

Since #651 the wrapper.household contract rows run the household
calculator instead of erroring at once, and with contract 2.0.0 the local
gates pass two hours. The 120-minute limit cancelled every attempt and the
catch-up re-staged it, so no candidate has finished its gates since 17
October.

Fixes #659

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… lease

#627 is not part of the cause: attempts with #651 but not #627 also timed
out. The four-hour staging lease is shorter than the new limit; #662 tracks
raising it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis merged commit 1473bf8 into main Oct 18, 2026
13 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merged at 6588f78 (merge commit above). Audit:

@MaxGhenis
MaxGhenis deleted the fix/uk-qualify-timeout branch October 18, 2026 23:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UK qualify job times out at 120 minutes since the household route runs, so no candidate finishes its gates

1 participant