Skip to content

Route around recorded quota exhaustion and hand partial work to the fallback on resume #389

Closed

Description

@datacas

Priority: P1. Currently blocks resolving REV-001 on PR #384 (#330) while Codex is out of quota; waiting for the quota to reset is a workaround, so this is not P0.

Problem and rationale. The runtime cannot transfer an interrupted stage to its fallback executor while keeping the primary executor's partial work. Observed on #330 / PR #384 (resolve stage, deep_coder, Codex out of quota):

  1. Recorded exhaustion is ignored when routing. After the runtime recorded Codex as exhausted until 2026-11-03 14:20 UTC, the next resume of the same stage still dispatched to Codex first and failed again.
  2. The reset date is dropped. The provider message said "Nov 4th, 2026 3:20 PM"; the runtime stored 14:20 UTC on the current day (Nov 2, then rolled to Nov 3 once past). Date and ordinal are lost and only the time survives.
  3. The advised command does not exist. The stop says to run --continue to use the configured fallback, but run_cycle.py rejects --continue together with --from resolve.
  4. The guards conflict. The fallback reroute is refused while the worktree has uncommitted changes (reroute stopped: dirty_workspace), and a resume is refused unless the worktree HEAD equals the PR head (the working directory is at …, not at …, the head commit of pull request 384). A temporary WIP commit satisfies the first guard and breaks the second, so an exhausted executor's partial work can never reach the fallback.

Objective and scope.

  • Routing skips an executor whose recorded exhaustion is still active, for resumed stages too, and says which executor it used instead.
  • The reset timestamp is parsed completely: date, year, time and timezone when available. When it is ambiguous, the original message is preserved, the uncertainty is recorded, and the runtime does not assume an earlier reset.
  • The stop message names a command the CLI accepts.
  • A controlled, auditable handoff for quota exhaustion: the PR HEAD stays exactly as it is, uncommitted changes are preserved, and no temporary WIP commits are created. The fallback continues on those changes. The runtime records which executor started the work and which continued it, checks provenance and diff integrity before the handoff, and prevents a second agent from overwriting existing modifications without inspecting them. The clean-worktree requirement stays for ordinary executor changes; the handoff is a controlled exception for quota exhaustion only.
  • The general rule that the worktree HEAD equals the PR head is not weakened.

Dependencies. None strict. Related and not a substitute: #285 (per-run routing overrides) and #262 (resumable budget stop). Unblocks the REV-001 resolution on #384.

Acceptance criteria.

  • A stage resumed while its primary executor is recorded exhausted dispatches to the fallback and reports the substitution and its reason.
  • The stored reset time equals the message's date, time and timezone, including ordinal days ("4th"); an ambiguous message keeps the original text and never produces an earlier reset.
  • The command named in the stop message is accepted by the CLI.
  • A resume whose worktree holds an exhausted executor's uncommitted changes reaches the fallback with those changes byte-for-byte intact and HEAD unchanged; the record names both executors and the diff fingerprint. An unrelated or altered diff stops the handoff with an accurate reason.
  • Ordinary executor changes with a dirty worktree are still refused.
  • .code-cycle.yml and sandbox settings are unchanged.

Regression tests.

  • Resume with an exhausted primary executor (dispatches to the fallback; no second Codex attempt).
  • Reset-time parsing: ordinals, past and future dates, timezone present and absent, ambiguous text.
  • The --continue / --from messaging.
  • Handoff with a dirty tree: the fallback receives the three modified files unchanged and continues with HEAD at the PR head; the same on an altered diff stops; ordinary dirty-tree changes still refused.
  • A fixture modelled on the Use the shared execution plan in supervised Orca cycles #330 / PR feat(orca): share execution plans in supervised cycles (#330) #384 state (resolve, deep_coder, three modified files).

Out of scope. Changing quota policy, routing profiles, or the HEAD-equals-PR-head guard.

Design decisions (refinement)

  1. Gate scope. The provenance gate applies to dirty --from resumes only. A clean-worktree --from resume is unaffected. A dirty --from resume proceeds only with a matching quota-exhaustion record for the stage's executor plus verifiable worktree provenance and diff integrity; otherwise it stops with an explicit reason and without discarding or modifying the worktree or HEAD. implement --continue keeps the behaviour A primary that runs out of quota over partial work can never fall back: --continue keeps routing to it #338 defined; Route around recorded quota exhaustion and hand partial work to the fallback on resume #389 does not change it. A quota-exhaustion record alone is not proof of ownership of the changes. The HEAD-equals-PR-head invariant is unchanged.

  2. Provenance and the legacy state of PR feat(orca): share execution plans in supervised cycles (#330) #384. New attempts record provenance automatically (a canonical SHA-256 over the complete diff, covering staged, unstaged and relevant untracked changes; the original and destination executors; stage; PR; HEAD). For worktrees that predate diff recording, such as the three-file dirty tree of PR feat(orca): share execution plans in supervised cycles (#330) #384, support a single, explicit, auditable operator attestation. It is bound to the exact PR, HEAD, stage, original executor, destination executor and fingerprint, and records the operator identity and timestamp. It requires explicit operator confirmation of the exact fingerprint before use, is single-use, and is invalid after any change to the diff. Its provenance is operator_attested, never executor_verified. It is never inferred from telemetry, never generated automatically, never reusable, and never counts as functional verification. It is not a general bypass for future unattested worktrees. Sandbox restrictions and the command policy are unchanged.

  3. Quota reset text and ambiguity. The original provider message is kept in the journal and the stop report only, with sensitive content redacted. Telemetry (quota_exhaustions) stays prose-free: a closed ambiguity token such as reset_time_ambiguous plus structured timestamp fields. An unambiguous reset is parsed with its complete date, year, time and timezone. An ambiguous reset takes the latest defensible interpretation, never an earlier one; 30 minutes is only a minimum safety bound, not a substitute for a later provider-stated reset. If no reliable interpretation exists, the executor stays unavailable until an explicit availability check or another authoritative reset signal, and no recovery timestamp is invented. Regression tests cover the original "Nov 4th, 2026 3:20 PM" message, ordinal dates, timezone ambiguity and an unresolvable date.

  4. How an uninterpretable reset ends. A quota message whose reset date is present but cannot be interpreted with confidence keeps the executor unavailable until an explicit operator command authorizes a retry. A message with no reset information at all keeps the conservative 30-minute window. The operator command targets an explicit executor and quota-exhaustion record, records the action and timestamp in the journal, preserves the historical exhaustion record and the original redacted message, and authorizes a new dispatch attempt without claiming the provider quota has recovered. If that attempt exhausts the quota again, a new exhaustion record is created. There is no arbitrary expiry (such as 24 hours), ambiguous records are never cleared silently, and the telemetry no-prose contract is unchanged. Tests cover an ambiguous reset, a missing reset and explicit operator retry authorization.

  5. Provenance outlives the quota window. Worktree provenance (original executor identity, stage, PR HEAD and diff fingerprint) and quota availability have independent lifecycles. A quota record expiring does not delete the provenance record; provenance is kept until the handoff or resume that uses it succeeds, and the historical audit record is never deleted. After the window expires, a dirty --from resume may return to the original executor when the provenance matches exactly; the work is not transferred to the fallback merely because an exhaustion record once existed. Expiry means retry eligibility, not proof that the provider quota has recovered. If the original executor exhausts its quota again, the new exhaustion is recorded and a controlled fallback handoff is allowed. Exact diff integrity and PR HEAD are validated, and any difference stops the resume without altering the worktree. Tests cover a resume before expiry, after expiry, repeated quota exhaustion and a fingerprint mismatch.

Activity

  1. added
    bugSomething isn't working
    P1Needed for 2.0, after the critical path
    on Nov 2, 2026
  2. added a commit that references this issue on Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Needed for 2.0, after the critical pathbugSomething isn't working

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions