Skip to content

feat(chat): requests waiting in the queue survive an isolate reset - #2918

Merged
mattzcarey merged 0 commit into
refactor/turn-core-rfcfrom
refactor/turn-core-durable-queue
Nov 20, 2026
Merged

mattzcarey merged 0 commit into
refactor/turn-core-rfcfrom
refactor/turn-core-durable-queue

Conversation

@mattzcarey

@mattzcarey mattzcarey commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

A client request that was waiting in the turn queue when the isolate reset now gets its own reply, after the turn the reset cut off. Each accepted request writes a small admission record before it stores its user message. The record goes when the request's turn has a fiber, or when the request ends without one. After a reset, once no interrupted turn waits for recovery, the records left are queued again in order, each under its own request id. Pending Think submissions wait for that recovery the same way.

The turn queue lives in memory and only the running turn has a fiber, so a deploy, eviction or crash used to lose every request that was waiting. Its user message stayed in the transcript with no reply of its own (U1, U2, A1 where it should end U1, U2, A1, A2), and its tab never got a terminal frame. Think had the same bug for one running and one pending submission: both ended completed with one reply between them.

What an app can observe:

  • After a reset, a request that was waiting runs as its own turn under its own request id. A tab that reconnects gets its stream, or its terminal frame.
  • A stop by the request id, before or after the reset, ends it aborted without calling the model. A clear deletes every record. Under latest, merge and debounce, a request a later one superseded stays superseded.
  • A waiting request's user message no longer counts as the conversation moving on when a recovery checks the leaf. So the recovered turn still continues or retries, where it used to skip.
  • A recovered turn that is retried (it had streamed nothing) still builds its request from the stored transcript. So it sees the questions waiting behind it, as a live turn does when they arrive before its model call.
  • Replays start after the app's onStart.
  • A replay ends skipped when its message is gone, or when a user message that nothing is waiting on follows it (for example after a rollback). A record older than an hour is dropped.
  • Think: a submission pending at the reset runs after the recovery of the running one. One accepted after the reset runs as before.
  • A new table, cf_agents_chat_admissions, costs three statements (two rows written) per client request. An older release never reads it.

The rule was model-checked first (ChatAdmission.tla, with a reset at every step and a broken variant for each guard). The ai-chat and Think tests reset the isolate for real with ctx.abort() and failed before this change. The core has unit tests on SQLite, and the agents, ai-chat and Think suites ran. On a deployed Worker (tc-queue), 100 of 100 rooms ended U1, U2, A1, A2 with the second tab getting its reply: 40 reset with ctx.abort() and 60 with a real wrangler deploy under code_update_strategy: immediate, half AIChatAgent and half Think. 16 of 16 submission pairs ended in order. Upgrade from the base release and rollback to it both behave like the base, with no errors.

@changeset-bot

changeset-bot Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 0c3797b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
agents Patch
@cloudflare/ai-chat Patch
@cloudflare/think Patch
@cloudflare/agent-think Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@pkg-pr-new

pkg-pr-new Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Open in StackBlitz

agents

npm i https://pkg.pr.new/agents@2918

@cloudflare/ai-chat

npm i https://pkg.pr.new/@cloudflare/ai-chat@2918

@cloudflare/codemode

npm i https://pkg.pr.new/@cloudflare/codemode@2918

hono-agents

npm i https://pkg.pr.new/hono-agents@2918

@cloudflare/shell

npm i https://pkg.pr.new/@cloudflare/shell@2918

@cloudflare/think

npm i https://pkg.pr.new/@cloudflare/think@2918

@cloudflare/voice

npm i https://pkg.pr.new/@cloudflare/voice@2918

@cloudflare/worker-bundler

npm i https://pkg.pr.new/@cloudflare/worker-bundler@2918

commit: 0c3797b

@agent-think

agent-think Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

🟡 agents import sizes: 1 entry point grew

Entry point Exports Largest gzip change Size now
🟡 agents/chat 2 resized, 5 new up to +967 B (+9.42%) 11 KiB
Changed exports (7)
Import Gzip change Size now
🟡 agents/chat#ChatTurns +967 B (+9.42%) 11 KiB
🟡 agents/chat#ChatRecoveryEngine +137 B (+1.41%) 9.7 KiB
🆕 agents/chat#ChatAdmissions — 3.2 KiB
🆕 agents/chat#createChatAdmissionStore — 3.1 KiB
🆕 agents/chat#chatRecoveryNames — 3 KiB
🆕 agents/chat#CHAT_ADMISSIONS_TABLE — 2.6 KiB
🆕 agents/chat#CHAT_ADMISSION_TTL_MS — 2.6 KiB
How this works

Each runtime export is bundled on its own, minified, and gzipped. Changes smaller than 100 B, or smaller than 1% and 1 KiB, are ignored. Growth over 10% or 5 KiB is marked 🔴. This report is informational and does not fail CI. The workflow artifact contains every measurement.

Compared 7314bb0d → 0c3797b6 · workflow run · reported by agent-think[bot]

devin-ai-integration[bot]

This comment was marked as resolved.

@mattzcarey
mattzcarey changed the base branch from refactor/turn-core-repair to refactor/turn-core-rfc November 20, 2026 08:47
@mattzcarey
mattzcarey added this pull request to stack #2869 November 20, 2026 08:47

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 new potential issues.

Devin Review

Comment on lines +16487 to +16495
recoverySettled: () => {
this._resumeAdmissions();
if (this._submissionsBehindRecovery.size > 0) {
void this.keepAliveWhile(() =>
this._queuePendingSubmissionRuns()
).catch((error) =>
console.error("[Think] failed to queue submissions:", error)
);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Queued work stranded after persisted replies

After a reset during onChatResponse, _chatRecoveryNames() blocks queued work on the interrupted chat Task. _settlePersistedChatTurn finishes recovery without invoking recoverySettled. Waiting requests and pending submissions remain blocked until another reset or client request triggers replay.

Learn more

Startup defers queued requests while an unfinished chat Task exists, and pending submissions return without running. If the interrupted turn already persisted its reply, _settlePersistedChatTurn replays the owed response hook and returns before entering the recovery engine. The engine therefore creates no incident and sends no settlement notification. The chat Task handler subsequently finishes, but nothing retries the deferred work. Startup deliberately leaves the owed hook for recovery, so it does not eliminate this case.

Example: Submission s1 persists its reply and awaits an application response hook while s2 remains pending. Reset the isolate during that hook. Recovery finishes s1's hook, but s2 remains pending without executing; an orphan WebSocket request behind s1 has the same problem.

Recommended fix: Notify durable admission and submission replay when this persisted-turn fast path finishes, at a boundary where the recovered Task no longer counts as pending recovery. Add a real-reset regression with an asynchronous response hook and both kinds of waiting work.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +532 to +538
await this.adapter.putIncident(key, incident);
if (
incident.status !== "detected" &&
incident.status !== "scheduled" &&
incident.status !== "attempting"
) {
this.#settled();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Queued turns overtake exhausted recovery persistence

When beginIncident() exhausts recovery, #settled() releases queued requests before the orphaned partial is persisted. chatRecoveryNames() ignores the original Task because its incident is terminal. Queued requests can call the model without the recovered text or settled tool results.

Learn more

beginIncident writes an exhausted incident before returning to the wake handler. The notification starts admission replay outside the recovery attempt, while #replayRecoveredTurn still has to persist the orphaned partial and terminalize recovery. chatRecoveryNames treats a run named by a terminal incident as no longer waiting, so the unfinished original Task does not prevent replay. A delayed application persistence hook makes the overlap directly observable.

Example: Configure maxInterruptedAttempts: 1, queue r2 behind r1, and reset while r1 has buffered reply content. Exhaustion writes the terminal incident and starts r2 while recovery is still persisting r1's partial. If persistence awaits external work, r2's model prompt lacks that partial and its settled tool results.

Recommended fix: Release waiting admissions only after orphan persistence and terminalization finish. Avoid notifying directly from every terminal incident write, or retain an explicit recovery-in-progress barrier until the entire wake handler settles. Test exhaustion with delayed partial persistence and assert that the queued model call starts afterward.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +527 to +528
// Requests an earlier isolate left go first.
if (admissions && !replay) await admissions.beforeEnqueue();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Older replays supersede newer client requests

During an active recovered turn, beforeEnqueue() restores older requests after a new overlapping request receives its sequence number. restore() allocates higher numbers to those older requests. Under latest, merge, or debounce, the newer request ends skipped instead of running.

Learn more

A fresh request calls decideSubmit before it persists messages or forces admission replay. On a fresh isolate, its overlapping sequence can therefore precede every restored orphan sequence. restore assigns increasing numbers in replay order, and isSuperseded treats those larger numbers as newer submissions. This reverses the relative age of restored and live requests.

Example: r2 waits behind interrupted r1 under latest. After reset, r1's recovered turn runs while r2 remains a record. A new r9 arrives and receives sequence 1 because r1 occupies the queue. Forced replay restores r2 with sequence 2, causing r9 to be skipped as superseded by r2.

Recommended fix: Reserve restored requests' concurrency ordering before allocating decisions for new client requests, or otherwise preserve their original ordering relative to live decisions. Add a regression that sends a new overlapping request while recovery is active and orphan records remain unreplayed.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

@mattzcarey
mattzcarey merged commit 0c3797b into main Nov 20, 2026
30 checks passed
@mattzcarey
mattzcarey deleted the refactor/turn-core-durable-queue branch November 20, 2026 10:53
@mattzcarey
mattzcarey restored the refactor/turn-core-durable-queue branch November 20, 2026 12:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant