From 20a34f28f3fc3a351601e5f699fb7aacb36bbc6b Mon Sep 17 00:00:00 2001 From: Matt Carey Date: Thu, 29 Oct 2026 16:45:00 +0000 Subject: [PATCH] [Agents] Split conversation state and memory into Sessions and Context concepts --- public/__redirects | 2 +- .../conversation-state-and-memory.mdx | 572 ------------------ src/content/docs/agents/concepts/memory.mdx | 82 +++ src/content/docs/agents/concepts/sessions.mdx | 77 +++ 4 files changed, 160 insertions(+), 573 deletions(-) delete mode 100644 src/content/docs/agents/concepts/conversation-state-and-memory.mdx create mode 100644 src/content/docs/agents/concepts/memory.mdx create mode 100644 src/content/docs/agents/concepts/sessions.mdx diff --git a/public/__redirects b/public/__redirects index 56f7907c1a9..c4c45bc1158 100644 --- a/public/__redirects +++ b/public/__redirects @@ -374,7 +374,7 @@ /agents/agentic-payments/mpp/ /agents/tools/payments/mpp/ 301 /agents/guides/ /agents/model-context-protocol/guides/ 301 /agents/concepts/agent-class/ /agents/agent-class/internals/ 301 -/agents/concepts/memory/ /agents/concepts/conversation-state-and-memory/ 301 +/agents/concepts/conversation-state-and-memory/ /agents/concepts/memory/ 301 /agents/getting-started/prompting/ /workers/get-started/prompting/ 301 /agents/platform/prompting/ /workers/get-started/prompting/ 301 /agents/platform/prompt.txt/ /workers/prompt.txt 301 diff --git a/src/content/docs/agents/concepts/conversation-state-and-memory.mdx b/src/content/docs/agents/concepts/conversation-state-and-memory.mdx deleted file mode 100644 index 534f3ded746..00000000000 --- a/src/content/docs/agents/concepts/conversation-state-and-memory.mdx +++ /dev/null @@ -1,572 +0,0 @@ ---- -title: Conversation state and memory -pcx_content_type: concept -description: How agents store and recall information, including read-only context, writable short-form memory, searchable knowledge, and on-demand skills. -tags: - - AI -sidebar: - order: 4 -products: - - agents ---- - -import { TypeScriptExample, LinkCard, WranglerConfig } from "~/components"; - -Agents need memory to be useful over time. Without it, every conversation starts from zero. The agent forgets who the user is, what it learned, and what it was doing. Memory is what turns a stateless LLM call into a persistent, context-aware agent. - -The [Session API](/agents/runtime/capabilities/sessions/) provides the memory layer for agents built on the Cloudflare Agents SDK. It manages two kinds of memory: **conversation history** (the messages and tool calls that make up a session) and **context memory** (persistent blocks injected into the system prompt that the agent can read, write, search, and load). - -Use this page when you need more than simple synced state or flat chat history. For small UI state, use [Store and sync state](/agents/agent-class/state/). For basic chat persistence, `AIChatAgent` stores messages for you. For opinionated long-term memory, Think builds on Session and context blocks. - -:::caution[Experimental] - -The Session memory APIs currently use `agents/experimental/memory/session`. The concepts are stable, but import paths and details may change before graduation. - -::: - -## Conversation history - -The most fundamental type of memory is the conversation itself: the messages between the user and the agent, the tool calls the agent made, and the results it received. The Session stores all of this in a tree-structured message history backed by a Session Provider, defaulting to SQLite. - - - -```ts -import { Session } from "agents/experimental/memory/session"; - -// Append messages as the conversation progresses -await session.appendMessage({ - id: `user-${crypto.randomUUID()}`, - role: "user", - parts: [{ type: "text", text: "What's the status of the deployment?" }], -}); - -// Read the full conversation history -const history = await session.getHistory(); -``` - - - -Conversation history persists across Durable Object hibernation and eviction. When the agent wakes up, the full history is available in SQLite. It does not need to be replayed or reconstructed. - -Messages are stored in a tree structure via `parent_id`, which enables branching conversations. When you `appendMessage` with a `parentId` that already has children, you create a branch, useful for features like response regeneration. `getHistory(leafId)` walks any chosen path through the tree. - -The Session also provides full-text search across the conversation history: - - - -```ts -const results = await session.search("deployment Friday", { limit: 10 }); -``` - - - -As conversations grow long, [compaction](#compaction) summarizes older messages to keep the context window manageable without losing the underlying data. - -## Context memory - -Context memory is persistent information injected into the system prompt, separate from the conversation history. It gives the agent access to identity, instructions, learned facts, knowledge bases, and reference material across every turn. - -The Session API supports four types of context memory, each suited to different kinds of information. The type is determined by the **provider** backing the context block. The Session detects the provider's capabilities automatically. - -### Read-only context - -This is your traditional system prompt: the agent's identity, personality, and instructions. You might write it directly in your codebase, load it from a `SOUL.md` file in R2, or fetch it from an API. The content is injected into the system prompt and the agent cannot modify it. - -A coding assistant might have a soul that defines its personality and constraints: - - - -```ts -import { Session } from "agents/experimental/memory/session"; - -const session = Session.create(this).withContext("soul", { - provider: { - get: async () => - "You are a senior TypeScript engineer. You write concise, " + - "well-tested code. You prefer composition over inheritance. " + - "When you are unsure, you say so rather than guessing.", - }, -}); -``` - - - -Or load it from R2 so you can update the agent's personality without redeploying: - - - -```ts -const session = Session.create(this).withContext("soul", { - provider: { - get: async () => { - const obj = await env.CONFIG_BUCKET.get("soul.md"); - return obj ? obj.text() : "You are a helpful assistant."; - }, - }, -}); -``` - - - -Read-only blocks are defined by providing an object with only a `get()` method. No tools are generated. The content appears in the system prompt and the agent has no way to change it. - -### Writable short-form context - -Think of this as a scratchpad the agent maintains for itself, a place to jot down things it needs to remember. Like how Claude Code keeps a todo list of tasks to work through, or how a customer support agent might track what it has learned about the user during the conversation. - - - -```ts -const session = Session.create(this) - .withContext("memory", { - description: "Important facts learned during conversation", - maxTokens: 1100, - }) - .withContext("todos", { - description: "Task list, track what needs to be done and what is complete", - maxTokens: 2000, - }); -``` - - - -When you omit the `provider` option in the builder, the Session auto-wires to a SQLite-backed writable provider. The agent gets a `set_context` tool that lets it replace or append content to these blocks. Token limits are enforced, so the agent cannot write more than the `maxTokens` budget allows. - -The system prompt renders writable blocks with a token usage indicator so the agent knows how much space it has left: - -```txt -══════════════════════════════════════════════ -MEMORY (Important facts learned during conversation) [45% — 495/1100 tokens] [writable] -══════════════════════════════════════════════ -User prefers dark mode. -User's project uses React and TypeScript. -Deployment target is Cloudflare Workers. - -══════════════════════════════════════════════ -TODOS (Task list) [12% — 240/2000 tokens] [writable] -══════════════════════════════════════════════ -- [x] Set up project scaffolding -- [ ] Add authentication middleware -- [ ] Write integration tests -``` - -The content persists across messages and survives hibernation. It is always visible in the system prompt, so the agent sees it on every turn without needing to fetch anything. - -### Searchable context - -When you have a large body of information (a knowledge base, documentation, logs, accumulated notes) you do not want to stuff it all into the system prompt. Searchable context keeps a summary in the system prompt (for example, "42 entries indexed") and lets the agent retrieve specific entries when it needs them. - -You provide a provider with a `search()` method. How that search works is entirely up to you: full-text search, vector search via [Vectorize](/vectorize/), a call to an external API, or anything else. The Session does not care about the implementation, only that the provider has a `search()` method. - -The built-in `AgentSearchProvider` uses Durable Object SQLite with FTS5 as default: - - - -```ts -import { AgentSearchProvider } from "agents/experimental/memory/session"; - -const session = Session.create(this).withContext("knowledge", { - description: - "Searchable knowledge base, search for relevant information before answering", - provider: new AgentSearchProvider(this), -}); -``` - - - -But you can implement your own provider backed by any search mechanism: - - - -```ts -const session = Session.create(this).withContext("knowledge", { - description: "Searchable knowledge base", - provider: { - get: async () => "Product documentation and FAQs", - search: async (query) => { - // Use Vectorize, an external API, whatever you need - const results = await env.VECTORIZE_INDEX.query( - await generateEmbedding(query), - { topK: 5 }, - ); - return results.matches.map((m) => m.metadata.text).join("\n\n"); - }, - set: async (key, content) => { - // Index new content - }, - }, -}); -``` - - - -The agent gets a `search_context` tool for querying and a `set_context` tool for indexing new entries. It decides what to search for, and you decide how the search works. - -This is the right choice when the agent needs to find specific pieces of information from a large collection, rather than loading entire documents. - -### Loadable context (Skills) - -Skills are large pieces of context (complete documents, reference guides, runbooks, templates) that the agent can discover and load on demand. Think of them as reference material on a shelf: the agent sees a list of titles and descriptions, picks what is relevant to the current task, loads it, uses it, and unloads it when done. - -Unlike searchable context where the agent retrieves small chunks from a larger collection, skills are designed to be loaded whole. When an agent loads a skill, it gets the entire document in its context window. - -Skills are backed by the `SkillProvider` interface. A skill provider has three methods: - -- **`get()`** returns a metadata listing (titles and descriptions) that appears in the system prompt -- **`load(key)`** fetches the full content of a specific skill -- **`set(key, content, description?)`** writes or updates a skill entry (optional) - -The system prompt shows available skills as a listing. The `[loadable]` tag tells the LLM that these entries are not inline. It needs to use a tool to access the full content: - -```txt -══════════════════════════════════════════════ -SKILLS [loadable] -══════════════════════════════════════════════ -- api-ref: API Reference documentation -- style-guide: Company style guide -- deploy-checklist: Production deployment checklist -``` - -The agent sees the titles, decides which skill is relevant to the current task, and uses `load_context` to pull the full content into its working context. When it is done, it uses `unload_context` to free the space. When the skill provider implements `set()`, the agent can also write back, updating existing skills or creating new ones. - -```txt -Agent sees: "- deploy-checklist: Production deployment checklist" -User asks: "Walk me through a production deployment" -Agent calls: load_context({ block: "skills", key: "deploy-checklist" }) -→ Full checklist content is loaded into the agent's working context -``` - -#### R2-backed skills - -The built-in `R2SkillProvider` stores skills in a Cloudflare R2 bucket. Each skill is an R2 object with optional custom metadata for descriptions. - - - -```ts -import { Session, R2SkillProvider } from "agents/experimental/memory/session"; - -const session = Session.create(this) - .withContext("soul", { - provider: { - get: async () => - [ - "You are a helpful assistant with access to skills.", - "When a user asks you to do something, check the SKILLS section", - "for a relevant skill and use load_context to load it.", - ].join("\n"), - }, - }) - .withContext("memory", { - description: "Learned facts", - maxTokens: 1100, - }) - .withContext("skills", { - provider: new R2SkillProvider(env.SKILLS_BUCKET, { prefix: "skills/" }), - }) - .withCachedPrompt(); -``` - - - -The `prefix` option scopes the provider to a subdirectory in the bucket. Skill keys in the metadata listing are shown without the prefix, so `skills/api-ref` becomes `api-ref` in the system prompt. - -Use `keys` to allowlist specific prefix-relative skills for `get()` and `load()`: - - - -```ts -new R2SkillProvider(env.SKILLS_BUCKET, { - prefix: "skills/", - keys: ["deploy-checklist", "api-ref"], -}); -``` - - - -Add an R2 bucket binding to your Wrangler configuration: - - - -```jsonc -{ - "r2_buckets": [ - { - "binding": "SKILLS_BUCKET", - "bucket_name": "my-agent-skills" - } - ] -} -``` - - - -Skills are regular R2 objects. Upload them through any R2 interface (the Wrangler CLI, the dashboard, or the Workers API): - -```sh -# Upload a skill from a file -wrangler r2 object put my-agent-skills/skills/style-guide --file ./docs/style-guide.md --content-type text/markdown -``` - -To add descriptions (shown in the metadata listing), set custom metadata on the R2 object: - - - -```ts -await env.SKILLS_BUCKET.put("skills/api-ref", content, { - customMetadata: { description: "API Reference documentation" }, -}); -``` - - - -#### Custom skill providers - -You can back skills with any storage by implementing the `SkillProvider` interface: - - - -```ts -import type { SkillProvider } from "agents/experimental/memory/session"; - -class DatabaseSkillProvider implements SkillProvider { - private db: D1Database; - - constructor(db: D1Database) { - this.db = db; - } - - async get(): Promise { - const rows = await this.db - .prepare("SELECT key, description FROM skills ORDER BY key") - .all(); - if (rows.results.length === 0) return null; - return rows.results - .map((r) => `- ${r.key}${r.description ? `: ${r.description}` : ""}`) - .join("\n"); - } - - async load(key: string): Promise { - const row = await this.db - .prepare("SELECT content FROM skills WHERE key = ?") - .bind(key) - .first(); - return row ? (row.content as string) : null; - } - - async set(key: string, content: string, description?: string): Promise { - await this.db - .prepare( - "INSERT INTO skills (key, content, description) VALUES (?, ?, ?) " + - "ON CONFLICT(key) DO UPDATE SET content = ?, description = ?", - ) - .bind(key, content, description ?? null, content, description ?? null) - .run(); - } -} -``` - - - -The Session detects the `load()` method via duck-typing and generates the appropriate tools automatically. - -#### Skills vs other memory types - -| Aspect | Skills | Writable context | Searchable context | -| -------------------- | ----------------------------------- | ------------------------ | ---------------------------------- | -| **In system prompt** | Metadata listing only | Full content | Summary count | -| **Access pattern** | Load whole document by key | Always visible | Search by query | -| **Best for** | Large documents, reference material | Short notes, preferences | Large collections of small entries | -| **Context cost** | Low (until loaded) | Proportional to content | Low (until searched) | -| **Agent writes?** | Optional (if `set` implemented) | Yes (via `set_context`) | Yes (via `set_context`) | - -The key distinction: skills are **lazy**. They cost nearly nothing in the system prompt until the agent decides it needs one. This makes them ideal for large reference material where only a subset is relevant to any given conversation. - -## How agents interact with memory - -The Session automatically generates tools based on the provider types of your context blocks. You pass these tools to your LLM alongside your own application-specific tools: - - - -```ts -const sessionTools = await session.tools(); -const allTools = { ...sessionTools, ...myApplicationTools }; - -const result = streamText({ - model: myModel, - system: await session.freezeSystemPrompt(), - messages: await convertToModelMessages(await session.getHistory()), - tools: allTools, -}); -``` - - - -### Generated tools - -The Session generates tools dynamically based on what provider types are present: - -| Tool | Generated when | What it does | -| -------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | -| **`set_context`** | Any writable, skill, or search block exists | Writes content to a named block. For writable blocks, replaces or appends. For skill/search blocks, writes a keyed entry. | -| **`load_context`** | Any skill block exists | Loads full content of a document by key into the agent's context. | -| **`unload_context`** | Any skill block exists | Frees context space by removing a previously loaded document. The document remains available for re-loading. | -| **`search_context`** | Any search block exists | Full-text search within a searchable block. Returns the top results ranked by relevance. | -| **`session_search`** | Using `SessionManager` | Searches across all sessions (cross-conversation search). | - -The tools include descriptions and parameter schemas that tell the LLM which blocks are available and what they are for. The agent decides when and how to use them based on the conversation. - -For the full tool signatures and all Session methods, refer to the [Session API reference](/agents/runtime/capabilities/sessions/). - -## The system prompt - -Context blocks are assembled into a structured system prompt with clear headers and metadata. Each block gets a labeled section with tags indicating its type and capacity: - -```txt -══════════════════════════════════════════════ -SOUL (Identity) [readonly] -══════════════════════════════════════════════ -You are a helpful coding assistant who speaks concisely. - -══════════════════════════════════════════════ -MEMORY (Important facts) [45% — 495/1100 tokens] [writable] -══════════════════════════════════════════════ -User prefers dark mode. -User's project uses React and TypeScript. - -══════════════════════════════════════════════ -KNOWLEDGE (Searchable knowledge base) [searchable] -══════════════════════════════════════════════ -12 entries indexed. - -══════════════════════════════════════════════ -SKILLS [loadable] -══════════════════════════════════════════════ -- api-ref: API Reference documentation -- style-guide: Company style guide -``` - -The tags (`[readonly]`, `[writable]`, `[searchable]`, `[loadable]`) tell the LLM what kind of interaction is possible with each block. Token budgets show the agent how much space remains in writable blocks, helping it manage its own memory. - -## Gotchas - -### Prompt caching - -LLM providers (Anthropic, OpenAI, and others) cache the system prompt prefix. When consecutive requests share the same system prompt, the provider can skip re-processing that prefix, reducing both latency and cost. Breaking the cache (by changing the system prompt) throws away this benefit. - -The Session API is designed to work with prompt caching: - -- **`freezeSystemPrompt()`** renders the system prompt from all context blocks on the first call, then returns the cached value on subsequent calls. The prompt does not change between turns, even when the agent writes to memory via `set_context`. -- **`withCachedPrompt()`** persists the frozen prompt to storage, so it survives Durable Object hibernation and eviction. When the agent wakes up, it loads the same prompt without re-fetching from all providers. - -When the agent uses `set_context` to update a writable block, the underlying provider is updated immediately (the data is saved), but the frozen system prompt is **not** re-rendered. The LLM sees the update on its next turn only if you explicitly call `refreshSystemPrompt()`, which you typically do between conversation turns, not mid-turn. - -This means the system prompt stays stable throughout a multi-step tool-use turn, preserving the provider's prefix cache across every step. - - - -```ts -const session = Session.create(this) - .withContext("soul", { - provider: { get: async () => "You are a helpful assistant." }, - }) - .withContext("memory", { description: "Learned facts", maxTokens: 1100 }) - .withCachedPrompt(); // Persist the frozen prompt across hibernation - -// During a conversation turn: -const system = await session.freezeSystemPrompt(); // Same value every call -const tools = await session.tools(); - -// ... agent calls set_context to update memory ... -// The frozen prompt is NOT changed, prefix cache stays warm - -// Between turns (optional, if you want the agent to see its own updates): -await session.refreshSystemPrompt(); -``` - - - -### Compaction - -Long conversations eventually exceed the LLM's context window. Compaction addresses this at two levels: **macro-compaction** summarizes ranges of older messages, and **micro-compaction** truncates individual messages that are too large. - -#### Macro-compaction - -Macro-compaction summarizes older messages, but it never deletes the originals. - -It uses **overlays**: a summary is stored in a separate table, keyed by the message range it covers. When `getHistory()` is called, overlays are applied transparently at read time. The compacted range is replaced by a synthetic summary message. The underlying messages remain in SQLite, preserving the full conversation for audit, search, and branching. - -```txt -Messages: [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] - ↓ compaction ↓ -Overlay: [1] [2] [SUMMARY of 3-7] [8] [9] [10] - ↑ tail protected -``` - -The key points: - -- **Non-destructive**, original messages are never deleted. The full conversation is always available in the database. -- **Iterative**, when the conversation grows again and triggers another compaction, the existing summary is passed to the LLM to update, not replaced from scratch. -- **Boundary-aware**, compaction boundaries are shifted to avoid splitting tool call / tool result pairs. -- **Configurable**, `protectHead` preserves the first N messages (usually the system context), and `tailTokenBudget` keeps the most recent messages intact. - - - -```ts -import { createCompactFunction } from "agents/experimental/memory/utils/compaction-helpers"; - -const session = Session.create(this) - .withContext("memory", { maxTokens: 1100 }) - .onCompaction( - createCompactFunction({ - summarize: (prompt) => - generateText({ model: myModel, prompt }).then((r) => r.text), - protectHead: 3, - tailTokenBudget: 20000, - minTailMessages: 2, - }), - ) - .compactAfter(100_000); // Auto-compact when token estimate exceeds threshold -``` - - - -Auto-compaction triggers after `appendMessage()` when the estimated token count exceeds the threshold. Compaction failure is non-fatal, the message is already saved. - -#### Micro-compaction - -Micro-compaction works at the individual message level rather than across ranges. It handles two problems: - -**Read-time truncation**: `truncateOlderMessages()` shortens tool outputs and long text in older messages before sending them to the LLM. Recent messages (last 4 by default) are kept intact. This operates on a copy, stored messages are not mutated. - - - -```ts -import { truncateOlderMessages } from "agents/experimental/memory/utils"; - -const history = await session.getHistory(); -const truncated = truncateOlderMessages(history); -// Pass truncated history to the LLM -``` - - - -**Row size enforcement**: when a message is persisted (typically an assistant message with large tool outputs), it is checked against the SQLite row size limit. Oversized tool outputs are replaced with a preview and a note suggesting the tool be re-run. This prevents individual messages from exceeding storage limits while preserving the conversation flow. - -## Related - - - - - - diff --git a/src/content/docs/agents/concepts/memory.mdx b/src/content/docs/agents/concepts/memory.mdx new file mode 100644 index 00000000000..db884b419c0 --- /dev/null +++ b/src/content/docs/agents/concepts/memory.mdx @@ -0,0 +1,82 @@ +--- +title: Context and memory +pcx_content_type: concept +description: Context is what an agent keeps in mind across turns, a scratchpad it reads and writes. Learn the kinds of context, how they relate to skills, and how changes reach the model without breaking the prompt cache. +tags: + - AI +sidebar: + order: 5 +products: + - agents +--- + +A [session](/agents/concepts/sessions/) records what was said. Context is what the agent keeps in mind while it talks: who it is, what it has learned about the user, what it is in the middle of doing. Think of it as a scratchpad next to the transcript. The agent reads it on every turn, and it can write to parts of it. + +The two are related but different. The transcript only grows, and old parts of it get summarized away. Context is current. When the agent learns that the user deploys on Fridays, that fact belongs in context, where it stays readable after compaction has turned the conversation it came from into one paragraph. + +## Blocks + +Context is made of labelled blocks. Each block appears in the agent's system prompt as its own section, so the model can tell its instructions from its notes. What sets blocks apart is who can change them. + +| Block | Who changes it | What the model sees | Typical use | +| ---------- | -------------- | ---------------------------------------- | ------------------------------------------------ | +| Read-only | You | The full text | Identity, instructions, project conventions | +| Writable | The agent | The full text and how much space is left | Facts about the user, a task list, working notes | +| Searchable | The agent, you | A summary, such as an entry count | A knowledge base too large for the prompt | + +A read-only block is the classic system prompt. It can come from your code, a file in R2, or an API, so you can change an agent's instructions without redeploying it. + +A writable block is the scratchpad proper. The agent gets a tool to replace or append to it, and the block has a token budget. Showing the model how full a block is lets it decide when to tidy its own notes, the way a person rewrites a todo list that has grown too long. + +A searchable block is for knowledge that does not fit. The prompt carries only a summary, and the agent searches for the entries it needs. How the search works is up to you: full-text search in the agent's own storage, vectors in [Vectorize](/vectorize/), or an external service. + +## Context and skills + +[Agent Skills](/agents/runtime/execution/agent-skills/) follow the same idea for instructions. The prompt carries a catalog of skill names and descriptions, and the agent loads a skill's full text only when a task calls for it. A large library of instructions then costs a few lines per skill until one is needed. + +The difference is direction and form. Skills are written by you, follow the [Agent Skills specification](https://agentskills.io), and can bring resources and scripts with them. Context blocks have no spec. They are plain text, and the agent writes many of them itself. A useful rule: if you would put it in a `SKILL.md` file, it is a skill; if the agent would jot it down, it is context. + +## How context reaches the model + +The system prompt sits at the start of every request, so it sits at the start of the [prompt cache](/agents/concepts/sessions/#prompt-caching-shapes-the-session). If the prompt changed every time a block did, every turn would start cold. Context avoids that by never rewriting what the model has already been sent. + +When a conversation starts, the agent renders its blocks into a system prompt and keeps that exact text. Every later request opens with it. Writing to a block saves the new value right away, but the system prompt stays as it was. + +Some changes the model should know about before the next refresh: today's date, the state of a workspace, a block another process updated. For those, the change goes after the cached prefix, as a short note next to the user message of the turn that noticed it. The session records which message carried the note, so every later request repeats it in the same place. The note is sent once and never moves, and each request still extends the one before it. + +```txt +system: snapshot of the blocks (unchanged for the whole conversation) +user: What should I work on? +assistant: ... +user: What day is it? + + note: environment changed, "Today is Tuesday." +assistant: It is Tuesday. +user: Thanks +``` + +Changes the agent makes itself need no note. The tool call that wrote the block is already in the transcript, so the model knows what it wrote. + +A refresh renders the blocks again and records a new snapshot. That is the one deliberate cache miss. It fits best where the cache is already gone, which is why agents refresh their context when a session is compacted. + +## Where context lives + +Block values live in the agent's storage or in whatever backs the block, outside the transcript. The session records only what the model was shown and where: the snapshot each stretch of the conversation started with, and each note delivered after it. The value is the agent's memory. The record is the history of what the model knew. + +That split lets you edit the memory directly without touching the conversation, and lets the conversation be replayed, branched, or compacted without losing the memory. + +## Choosing where information goes + +| Information | Where it belongs | +| ----------------------------------------------- | -------------------------------------------------- | +| What the user and the agent said | The [session](/agents/concepts/sessions/) | +| Who the agent is and how it behaves | A read-only block | +| What the agent learned and wants to keep | A writable block | +| A large collection the agent looks things up in | A searchable block | +| Reusable instructions loaded only when needed | A [skill](/agents/runtime/execution/agent-skills/) | +| Application data a client displays and edits | [Agent state](/agents/agent-class/state/) | + +## Related + +- [Sessions](/agents/concepts/sessions/) covers the transcript, branching, compaction, and prompt caching. +- [Agent Skills](/agents/runtime/execution/agent-skills/) is the reference for on-demand instructions. +- [Think](/agents/harnesses/think/) declares context blocks for you and delivers their changes each turn. diff --git a/src/content/docs/agents/concepts/sessions.mdx b/src/content/docs/agents/concepts/sessions.mdx new file mode 100644 index 00000000000..ac435767bf6 --- /dev/null +++ b/src/content/docs/agents/concepts/sessions.mdx @@ -0,0 +1,77 @@ +--- +title: Sessions +pcx_content_type: concept +description: A session is the durable record of one conversation. Learn how the transcript forms a tree, how compaction shortens what the model sees without deleting anything, and why prompt caching shapes both. +tags: + - AI +sidebar: + order: 3 +products: + - agents +--- + +A session is the record of one conversation: every message the user sent, every reply, and every tool call and result, in order. An agent keeps it in its own storage, so the conversation survives hibernation, eviction, and deploys. When the agent wakes up, the history is already there. + +The session answers one question: what happened? It does not decide what the agent should remember about the user or what instructions it follows. That is [context and memory](/agents/concepts/memory/), a separate concept that sits next to the session. + +## The transcript is a tree + +Each message points to the message before it. Most of the time that makes a straight line. When a user regenerates a reply or edits an earlier message, the new message points to the same parent as the old one, and the conversation branches. + +```txt +user: Plan a trip to Lisbon +└─ assistant: Here is a 3-day plan... + └─ user: Make it 5 days + ├─ assistant: Here is a 5-day plan... (first attempt) + └─ assistant: A 5-day plan, slower pace... (regenerated, latest) +``` + +Nothing is overwritten. The first attempt is still stored, and a client can show both. The active branch is the path from the first message to the latest leaf, and that path is what the model sees. Pick another leaf and you get another conversation from the same history. + +## The record and the request + +The session stores everything. The model never gets everything. + +Each time the agent calls a model, it builds a request from one branch of the transcript: a system prompt, then the messages on that path. That request is a projection of the record, not a copy of it. It can replace old messages with a summary, shorten large tool outputs, or swap images from older turns for a placeholder, while the stored messages stay as they were. + +Keeping the two apart is what lets a conversation outgrow the model's context window without losing anything. The record is for audit, search, branching, and showing the conversation to people. The request is for the model, and it only has to fit the model. + +## Compaction + +A long conversation eventually needs a shorter request. Compaction summarizes a range of older messages and stores the summary as an overlay on that range. When the agent reads the branch, the overlay stands in for the messages it covers. + +```txt +Stored: [1] [2] [3] [4] [5] [6] [7] [8] [9] +Projected: [1] [2] [summary of 3-6] [7] [8] [9] +``` + +The original messages stay in storage, so search and branching still see them. The first few messages and the most recent ones stay verbatim: the opening says what the conversation is about, and the tail is what the model needs to answer next. A later compaction updates the existing summary instead of starting again, and the range never splits a tool call from its result. + +Compaction costs something every time it runs, which the next section explains. Good compaction is rare and large, not frequent and small. + +## Prompt caching shapes the session + +Model providers cache the beginning of a request. When a request starts with exactly the same bytes as the previous one, the provider skips the work for that prefix, and the call is faster and cheaper. On a long conversation, most of every request is that shared prefix. + +A session that only grows at the end keeps that cache warm. Each turn adds messages after the last one, so each request extends the request before it. Anything that changes earlier bytes breaks the cache from that point on: + +- changing the system prompt +- compacting a range into a summary +- shortening or removing an earlier message in the projection + +None of these are wrong. Some are necessary. But each one should happen deliberately and seldom, and the projection should produce the same bytes for the same history every time. This rule is also why context changes, such as an updated memory, are delivered after the cached prefix instead of rewriting the system prompt. [Context and memory](/agents/concepts/memory/) explains how. + +## Search + +Because the record keeps every message, an agent can search its own history: what did the user say about the deployment last week? Search covers messages that compaction has summarized, because the originals are still stored. + +## One conversation, one session + +A session holds one conversation. For chat applications, give each conversation its own agent instance, so conversations hibernate, scale, and fail independently. Use more than one session inside an agent when one conversation has drafts or side threads of its own. + +Harnesses differ in who owns the session. [Think](/agents/harnesses/think/) and `AIChatAgent` store their transcripts in the [Sessions capability](/agents/runtime/capabilities/sessions/). [Pi](/agents/harnesses/pi/) and [OpenCode](/agents/harnesses/opencode/) keep their own. If you build your own loop, the Sessions capability gives you the tree, compaction, and search described here. + +## Related + +- [Context and memory](/agents/concepts/memory/) covers what the agent keeps in mind across turns. +- [Sessions capability](/agents/runtime/capabilities/sessions/) is the API reference for storing a session in a Durable Object.