Repository navigation
feat(chat): requests waiting in the queue survive an isolate reset - #2918
Conversation
🦋 Changeset detectedLatest commit: 0c3797b The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
agents
@cloudflare/ai-chat
@cloudflare/codemode
hono-agents
@cloudflare/shell
@cloudflare/think
@cloudflare/voice
@cloudflare/worker-bundler
commit: |
🟡 agents import sizes: 1 entry point grew
Changed exports (7)
How this worksEach runtime export is bundled on its own, minified, and gzipped. Changes smaller than 100 B, or smaller than 1% and 1 KiB, are ignored. Growth over 10% or 5 KiB is marked 🔴. This report is informational and does not fail CI. The workflow artifact contains every measurement. Compared |
| recoverySettled: () => { | ||
| this._resumeAdmissions(); | ||
| if (this._submissionsBehindRecovery.size > 0) { | ||
| void this.keepAliveWhile(() => | ||
| this._queuePendingSubmissionRuns() | ||
| ).catch((error) => | ||
| console.error("[Think] failed to queue submissions:", error) | ||
| ); | ||
| } |
There was a problem hiding this comment.
🔴 Queued work stranded after persisted replies
After a reset during onChatResponse, _chatRecoveryNames() blocks queued work on the interrupted chat Task. _settlePersistedChatTurn finishes recovery without invoking recoverySettled. Waiting requests and pending submissions remain blocked until another reset or client request triggers replay.
Learn more
Startup defers queued requests while an unfinished chat Task exists, and pending submissions return without running. If the interrupted turn already persisted its reply, _settlePersistedChatTurn replays the owed response hook and returns before entering the recovery engine. The engine therefore creates no incident and sends no settlement notification. The chat Task handler subsequently finishes, but nothing retries the deferred work. Startup deliberately leaves the owed hook for recovery, so it does not eliminate this case.
Example: Submission s1 persists its reply and awaits an application response hook while s2 remains pending. Reset the isolate during that hook. Recovery finishes s1's hook, but s2 remains pending without executing; an orphan WebSocket request behind s1 has the same problem.
Recommended fix: Notify durable admission and submission replay when this persisted-turn fast path finishes, at a boundary where the recovered Task no longer counts as pending recovery. Add a real-reset regression with an asynchronous response hook and both kinds of waiting work.
Was this helpful? React with 👍 or 👎 to provide feedback.
| await this.adapter.putIncident(key, incident); | ||
| if ( | ||
| incident.status !== "detected" && | ||
| incident.status !== "scheduled" && | ||
| incident.status !== "attempting" | ||
| ) { | ||
| this.#settled(); |
There was a problem hiding this comment.
🔴 Queued turns overtake exhausted recovery persistence
When beginIncident() exhausts recovery, #settled() releases queued requests before the orphaned partial is persisted. chatRecoveryNames() ignores the original Task because its incident is terminal. Queued requests can call the model without the recovered text or settled tool results.
Learn more
beginIncident writes an exhausted incident before returning to the wake handler. The notification starts admission replay outside the recovery attempt, while #replayRecoveredTurn still has to persist the orphaned partial and terminalize recovery. chatRecoveryNames treats a run named by a terminal incident as no longer waiting, so the unfinished original Task does not prevent replay. A delayed application persistence hook makes the overlap directly observable.
Example: Configure maxInterruptedAttempts: 1, queue r2 behind r1, and reset while r1 has buffered reply content. Exhaustion writes the terminal incident and starts r2 while recovery is still persisting r1's partial. If persistence awaits external work, r2's model prompt lacks that partial and its settled tool results.
Recommended fix: Release waiting admissions only after orphan persistence and terminalization finish. Avoid notifying directly from every terminal incident write, or retain an explicit recovery-in-progress barrier until the entire wake handler settles. Test exhaustion with delayed partial persistence and assert that the queued model call starts afterward.
Was this helpful? React with 👍 or 👎 to provide feedback.
| // Requests an earlier isolate left go first. | ||
| if (admissions && !replay) await admissions.beforeEnqueue(); |
There was a problem hiding this comment.
🟡 Older replays supersede newer client requests
During an active recovered turn, beforeEnqueue() restores older requests after a new overlapping request receives its sequence number. restore() allocates higher numbers to those older requests. Under latest, merge, or debounce, the newer request ends skipped instead of running.
Learn more
A fresh request calls decideSubmit before it persists messages or forces admission replay. On a fresh isolate, its overlapping sequence can therefore precede every restored orphan sequence. restore assigns increasing numbers in replay order, and isSuperseded treats those larger numbers as newer submissions. This reverses the relative age of restored and live requests.
Example: r2 waits behind interrupted r1 under latest. After reset, r1's recovered turn runs while r2 remains a record. A new r9 arrives and receives sequence 1 because r1 occupies the queue. Forced replay restores r2 with sequence 2, causing r9 to be skipped as superseded by r2.
Recommended fix: Reserve restored requests' concurrency ordering before allocating decisions for new client requests, or otherwise preserve their original ordering relative to live decisions. Add a regression that sends a new overlapping request while recovery is active and orphan records remain unreplayed.
Was this helpful? React with 👍 or 👎 to provide feedback.
A client request that was waiting in the turn queue when the isolate reset now gets its own reply, after the turn the reset cut off. Each accepted request writes a small admission record before it stores its user message. The record goes when the request's turn has a fiber, or when the request ends without one. After a reset, once no interrupted turn waits for recovery, the records left are queued again in order, each under its own request id. Pending
Thinksubmissions wait for that recovery the same way.The turn queue lives in memory and only the running turn has a fiber, so a deploy, eviction or crash used to lose every request that was waiting. Its user message stayed in the transcript with no reply of its own (
U1, U2, A1where it should endU1, U2, A1, A2), and its tab never got a terminal frame.Thinkhad the same bug for one running and one pending submission: both endedcompletedwith one reply between them.What an app can observe:
abortedwithout calling the model. A clear deletes every record. Underlatest,mergeanddebounce, a request a later one superseded stays superseded.onStart.skippedwhen its message is gone, or when a user message that nothing is waiting on follows it (for example after a rollback). A record older than an hour is dropped.Think: a submission pending at the reset runs after the recovery of the running one. One accepted after the reset runs as before.cf_agents_chat_admissions, costs three statements (two rows written) per client request. An older release never reads it.The rule was model-checked first (
ChatAdmission.tla, with a reset at every step and a broken variant for each guard). The ai-chat and Think tests reset the isolate for real withctx.abort()and failed before this change. The core has unit tests on SQLite, and the agents, ai-chat and Think suites ran. On a deployed Worker (tc-queue), 100 of 100 rooms endedU1, U2, A1, A2with the second tab getting its reply: 40 reset withctx.abort()and 60 with a realwrangler deployundercode_update_strategy: immediate, halfAIChatAgentand halfThink. 16 of 16 submission pairs ended in order. Upgrade from the base release and rollback to it both behave like the base, with no errors.