Repository navigation
🤖 feat: Daytona sandboxes as a workspace runtime - #7100
ibetitsmike wants to merge 162 commits into
Conversation
…sh config Part A of the Daytona runtime provider (ssh + daytona sub-config): - runtime schema: optional `daytona` sub-config on the ssh variant, mutually exclusive with `coder`; `daytona` runtime enablement ID (UI-only, like coder) and a shared enablement-ID -> RuntimeMode record. - shared size presets (small/medium/large) in src/common/constants/daytona.ts. - credentials in <xum home>/daytona.json (0600, atomic), with DAYTONA_API_KEY / DAYTONA_API_URL fallbacks; the UI only sees a masked view. - `daytona` oRPC namespace: getConfig, setConfig, testConnection, listSandboxes, listSnapshots. - thin typed fetch client with timeouts and typed errors (no @daytona/sdk). - DaytonaService: labels with the installation identity, per-sandbox SSH token cache with refresh-then-revoke, startAndWait with bounded polling. - daytonaSshConfig: 0600 include file with one Host block per sandbox, gateway known_hosts, a managed `Host *.xum--daytona` + Include block in ~/.ssh/config (shared marker-block writer extracted from the Coder writer), and an `ssh -G` check that the token is the resolved user. - sshConfigParser (ssh2 path) now resolves Include directives.
…parsing into small units
…and wiring Part B of the Daytona runtime provider (ssh + daytona sub-config): - DaytonaSSHRuntime (mirrors CoderSSHRuntime): finalizeConfig names a new sandbox before persist (Daytona takes a name wherever it takes an id) or attaches a given one as shared; validateBeforePersist checks credentials, preset size and name/attach existence; postCreateSetup creates with preset resources and lifecycle intervals, falls back to create-then-resize on HTTP 400, waits, writes ssh access, seeds the agent home and deletes the sandbox it created on failure; ensureReady caches a ready result for 60 s per sandbox, shares one in-flight start, reports "Starting sandbox" and maps a missing sandbox to runtime_not_ready; forkWorkspace marks both shared; deleteWorkspace starts a stopped sandbox so the dirty/unpushed checks run. - DaytonaService: create/resize/stop/delete/refreshActivity, single-flight ensureSandboxReady, ssh include ownership with token refresh, ready cache. - Agent-home seeding of tool_env/tool_pre/tool_post and global skills to the paths remote lookups read (~/.mux, ~/.agents/skills), never for attached boxes. - runtimeFactory selects DaytonaSSHRuntime; checkRuntimeAvailability reports Daytona availability; bash AI proxy forwards are not restored at startup. - Archive reuses the Coder keep/stop/delete behavior for dedicated sandboxes. - DaytonaReconciler (service container, 60 s, never blocks startup): keepalive for active workspaces, token refresh, stop-only orphan sweep with archived workspaces counted as references, per-call timeouts. - Gated TEST_DAYTONA=1 real test (skipped by default).
…er the CRAP limit
…e in archive refusals, report sandbox states Archive with the delete behavior now runs the same dirty and unpushed checks as deleteWorkspace on a dedicated Daytona sandbox (starting a stopped one for them). When they fail, the sandbox is stopped instead and the archive result carries a warning the sidebar and menu bar show. Archive refusals name the Daytona sandbox instead of a Coder workspace, and getRuntimeStatuses reports Daytona sandbox states (starting, running, stopped, archived, error).
…ontrols Settings > Runtimes gets a Daytona row: a global access form (write-only API key in its own file, API URL, target, test connection) and a project-scope defaults form (snapshot and size preset, saved to lastRuntimeConfig.ssh.daytona). Workspace creation gets a Daytona runtime button with New/Existing sandbox, snapshot and size controls, prefilled from the project defaults. Runtime badges and the footer chip show the sandbox state. Stories cover the forms at desktop and 390px widths as play-only contracts (Pixel budget has no headroom).
…g a shared-checkout child Side chats and isolation:none sub-agents copy the parent's runtime config, sandbox ID included, without the shared flag, so archiving one stopped or deleted the parent's sandbox.
…ook path The hooks page said remote user-level hooks live in ~/.xum; SSH remotes read ~/.mux and Docker reads /var/mux (RemoteRuntime/DockerRuntime getXumHome).
… and docs index Rebuild the runtime-choice context inside the auto-switch effect so it depends on its fields, not a per-render object; format; sync the xum-docs skill index.
Normalizing the project's default runtime in render broke the compiler's memoization, so selectedRuntime changed identity on every keystroke (useDraftWorkspaceSettings.compiler.test.tsx). The config type already admits daytona, so the normalization was unnecessary.
…RAP limits Extract the SSH badge, runtime-choice table, Coder/Daytona creation validation, Daytona access hooks and components, archive result helpers, the remembered-SSH merge, the delete-check exit-code mapping and the Daytona status probe. The creation form now loads Daytona lists itself while mounted. Covers the archive warning popover path with a test.
…s from starting sandboxes - Background git and PR status wait for a Daytona sandbox to run, as for devcontainers, so opening a workspace no longer starts a stopped sandbox. - RuntimeStatusStore refreshes when a turn starts a stopped runtime and when that start ends (runtime-status events from the workspace chat stream). - The API key status sits beside its label instead of inside it.
…clear errors - An invalid API URL throws DaytonaConfigError, which daytona.setConfig returns as BAD_REQUEST with its reason instead of Internal Server Error. - Client errors drop HTML tags, collapse whitespace and keep 200 chars of the server message.
…e findings - Delete refuses a dirty worktree under the unresolved ~/... path: checkWorkspaceDeletable expands a leading ~ remotely (also fixes Coder), and Daytona resolves the path first and fails closed when it cannot. - Daytona aliases keep their pinned gateway key: the headless OpenSSH override skips *.xum--daytona, and ssh2 checks the server key against the pinned daytona_known_hosts entries. - Daytona ssh tokens are redacted (<daytona-token>) from ssh stderr, errors, init output, pool probe errors and reverse-forward logs. - A fork reuses its source's sandbox (`forked`) without marking either workspace attached; the last referencing workspace, archived ones included, deletes it. Attached sandboxes are never deleted; their last workspace only releases Xum's ssh access. Archive skips a sandbox another workspace references. - A turn's runtime restarts a sandbox that stopped within the ready cache and retries the command once; transport failures invalidate the cache. - Tokens are revoked on settings changes and failed installs; a failed ssh -G user check reuses the token on retry. - Later turns after a failed setup report the setup failure and its reason. - Keepalive checks whether background jobs still run, skips stopped or unknown sandboxes, and skips the sweep when the sandbox list fails.
`new URL` reads "http:///api" and "https:api" as host "api", so they were saved as http://api. normalizeApiUrl now requires a written host and throws DaytonaConfigError ("has no host") otherwise.
…ings - Removing the last workspaces on one sandbox at the same time deletes it exactly once: the fate decision runs under a per-sandbox lock, and a removal that passed its dirty check is recorded so the next one does not count it. A refused (dirty) removal is never recorded and keeps the sandbox. - Startup loads every token from daytona.conf into the redactor and the token cache (due for refresh), so ssh errors after a restart never show it, and the next mint, a settings change or an archive revokes it. - When the gateway refuses a token (revoked outside Xum), the runtime mints a new one, rewrites the include file, resets the pool backoff and retries the command once. - A forced delete whose sandbox DELETE fails returns a removal warning that names the sandbox to delete by hand. - Delete waits (up to 60 s) for a sandbox Daytona is already destroying instead of failing with "is in state destroying".
…token A stopped or deleted sandbox also answers "Permission denied", so a command sent within the ready cache after a stop minted and revoked a token for nothing and then failed, and the first command after a delete outside Xum showed the refusal instead of "no longer exists". The runtime now reads the sandbox state first on an ssh failure: a stopped sandbox takes the restart path (turn runtimes), a missing one reports that it no longer exists, and only a started sandbox gets a new token.
…l on their claimed breaks - await the background-runtime rejection so its no-start assertion runs after the command settles - assert both concurrent ensureReady checks report the shared start (the old test passed without the dedupe) - assert the clean-delete test runs the SSH dirty check - round-trip snapshot, target and forked through RuntimeConfigSchema
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
Security findingsAdvisory findings (4)
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
🛡️ Codex Security Review · Automatically triggered
Here are some automated security review suggestions for this pull request.
Reviewed commit: 26fb323354
ℹ️ About Codex security reviews in GitHub
This is an experimental Codex feature. Security reviews are triggered when:
- You comment "@codex security review"
- A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review
Once complete, Codex will leave suggestions, or a comment if no findings are found.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 26fb323354
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
setConfig revoked only cached tokens; a mint in flight finished afterwards and wrote and cached an old-account token. A credentials generation is captured at mint start: a mint that spans a change revokes its token with the client that minted it, never writes or caches it, and mints again for the waiting callers.
…ours A name taken between validateBeforePersist and the create made Daytona refuse the create, and the cleanup then deleted the other install's sandbox. A failed create now deletes only a sandbox carrying this install's and workspace's labels; a successful create is still deleted when a later setup step fails.
The 256-token cap evicted tokens that were still live. Tokens now expire by age only.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3a991bd601
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b805476db8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… attached workspace is removed
… replaced When another backend sharing the root wrote a sandbox's first include block after this process's startup load, ensureSshAccess minted a replacement with no previous token, so the other backend's token stayed live until it expired. It is now held as due at once for the sandbox this process is about to use, so the mint replaces and revokes it. The other backend's next command is refused and goes through the refused-token path, which mints its own.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 10d01df2bb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…s: the current setting The archive hook's attached-last path also reads the setting when the last use ends, so removal and archive stay consistent; the docs say so.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8524ac758e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
🛡️ Codex Security Review · Automatically triggered
Here are some automated security review suggestions for this pull request.
Reviewed commit: ecebbfd6b5
ℹ️ About Codex security reviews in GitHub
This is an experimental Codex feature. Security reviews are triggered when:
- You comment "@codex security review"
- A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review
Once complete, Codex will leave suggestions, or a comment if no findings are found.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ecebbfd6b5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
An editor's Remote-SSH reads ~/.ssh/config without Xum's -F, and OpenSSH keeps the first value it reads, so an earlier user `Host *` that turns off host key checking, forwards the agent or sets a proxy applied to sandbox aliases. The Daytona managed block now goes on the first line (an appended block from an earlier Xum moves there), ends with a `Host *` scope reset so the user's following top-level lines still apply to every host, and each alias sets ProxyCommand none, ForwardAgent no, ForwardX11 no and PermitLocalCommand no. KnownHostsCommand is left alone: it needs OpenSSH 8.5 or an IgnoreUnknown that would replace the user's own.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4040f70bea
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ection is live While Xum reconnects, useAPI() can keep the last client, whose calls fail at once with ConnectionLostError, and the same object may come back. The creation controls' resources and the Settings access form loaded once per client, so a load that failed in the gap stayed failed. Both now load on useLiveAPI(): nothing runs while reconnecting, and the load runs again when the connection is live. The access form also clears its load error after a load that succeeds.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 69eb98671d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const ready = await args.api.workspace.ensureDaytonaSandboxReady({ | ||
| workspaceId: args.workspaceId, | ||
| }); |
There was a problem hiding this comment.
Admit editor use before starting the Daytona sandbox
When an archive races an editor click, this await starts or refreshes Daytona before recordEditorOpen acquires the editor workspace-use lease or checks archivingWorkspaces. For example, a stop archive can observe an already-stopped sandbox and decide no stop is needed after this endpoint has read the unarchived workspace but before it starts the sandbox; the subsequent recording is refused once the archive is durable, yet the archived sandbox remains running and billable. Native-terminal opening avoids this by holding its use lease before ensureDaytonaSandboxUp; move Daytona preparation into the same admitted backend operation as editor recording.
AGENTS.md reference: AGENTS.md:L170-L170
Useful? React with 👍 / 👎.
| filePath: this.filePath, | ||
| error: getErrorMessage(error), | ||
| }); | ||
| return {}; |
There was a problem hiding this comment.
Do not switch accounts after credential read errors
When daytona.json selects account A but reading it transiently fails due to a permission or I/O error, and the process also has DAYTONA_API_KEY/DAYTONA_API_URL for account B, returning {} makes resolve() silently choose B. The strict read added for applyUpdate() does not cover this runtime path, so the next account lease can pin B and run create, start, or delete operations there even though the saved settings still select A. Propagate non-ENOENT read failures instead of treating them as absent credentials.
Useful? React with 👍 / 👎.
| log.info("Daytona reconcile: stopping an unreferenced sandbox", { sandboxId: sandbox.id }); | ||
| try { | ||
| await this.call( | ||
| this.options.daytonaService.stopSandbox(sandbox.id), |
There was a problem hiding this comment.
Settle owned sandboxes after shared-child removals
When a Daytona parent is removed while an active taskIsolation: "none" child borrows its checkout, WorkspaceService skips the parent's runtime deletion to preserve the child; removing that child later also skips runtime deletion because it owns no checkout. Once the child record disappears, this orphan path only stops the sandbox, and because auto-delete was disabled at creation no remaining workspace can ever call deleteSandbox, leaving the Xum-owned sandbox's billed storage indefinitely. This is independent of the known two-backend last-reference race because both skips occur in the normal single-backend removal flow; defer ownership cleanup until the borrower ends or explicitly settle the sandbox when the last borrowed reference is removed.
Useful? React with 👍 / 👎.
Summary
Adds Daytona Cloud as a workspace runtime. A Daytona workspace is an
sshworkspace with adaytonasub-config: Xum creates (or attaches to) a sandbox through Daytona's REST API and reaches it through Daytona's SSH gateway, so bash, file tools, git, background jobs, terminals, and sub-agents run in the sandbox instead of on the Xum host. This is Phase 1b of the "move agent compute off the Xum host" plan; Phase 0 (cgroup priority) and Phase 2 (snapshots, node-side project defaults, resize) are separate follow-up PRs.Background
Default runtimes run every agent command on the Xum host, so builds and test suites compete with the server and the agent loop. The
sshruntime already moves all of that to a remote box (Coder issshplus acodersub-config), but no provider hands out a fresh, sized box in seconds. Building onSSHRuntimereuses exec, file ops, bundle sync, worktrees, fork, PTY, background jobs, and the AI proxy forward, plus every check keyed onssh, instead of adding a new runtime type.What the user gets
DAYTONA_API_KEY), API URL (for BYOC), target region, Test connection, Remove key.docs/runtime/daytona.mdx; also fixes the remote user-hook path indocs/hooks/tools.mdx(~/.mux, not~/.xum).Implementation
src/node/config/daytonaCredentialsStore.ts<xum home>/daytona.json, mode 0600, atomic write. The key never reaches the frontend, runtime config,providers.jsonc, orsecrets.json(which is injected into agent turns and terminals).src/node/services/daytona/daytonaClient.ts@daytona/sdkwas rejected: 2 MB unpacked and it pulls the AWS S3 client, OpenTelemetry, axios, and socket.io.src/node/services/daytona/daytonaService.tsxum.instance,xum.workspaceId,xum.project), token cache with refresh before expiry and revoke on replace/delete/settings change, start-and-wait with state mapping.src/node/runtime/DaytonaSSHRuntime.tsSSHRuntimeand mirrorsCoderSSHRuntime: create (snapshot or image, 30 min auto-stop, 1 day auto-archive, no auto-delete),ensureReady(start stopped/archived sandboxes, 60 s cache), one-shot recovery for a refused token, fork as shared, delete with dirty/unpushed checks (starting a stopped sandbox to run them) and last-reference deletion.src/node/runtime/daytonaSshConfig.ts<xum home>/ssh/daytona.conf(0600) with oneHost <id>.xum--daytonablock per sandbox; the token is the SSH user, so it stays out of argv and persisted config. Included from a managedHost *.xum--daytonablock in~/.ssh/config; anssh -Gcheck reports a userHost *Userline that would shadow the token.GET {apiUrl}/configover TLS intodaytona_known_hosts, withStrictHostKeyChecking yes. Every OpenSSH call for a sandbox also passes-o StrictHostKeyChecking=yes, so a userHost *entry above Xum's block cannot weaken it, and thessh -Gcheck reports a shadowedUserorUserKnownHostsFile. The SSH2 transport enforces the same pin, including underxum server.sshConfigParser.tsnow followsInclude, gated by the enclosingHostblock as OpenSSH does.src/node/runtime/daytonaTokenRedaction.ts<daytona-token>in SSH errors, logs, and terminal output.src/node/services/daytona/daytonaReconciler.tssrc/node/runtime/daytonaAgentHomeSeed.tstool_env,tool_pre,tool_post, and global skills (directories with a valid skill name and aSKILL.md) into a created sandbox's agent home, so global hooks run there. Follows a link inside a skill only while it stays in that skill. Never touches attached sandboxes.runtimeFactory.tsselectsDaytonaSSHRuntime;sshForwardTarget.tsskips restore at startup (no billed starts); archive reuses the remote-box setting (keep,stop, ordelete), falling back to stop when the delete path finds unsaved work; UI status checks (git, PR) never start a sandbox.Validation
/api-json, server v0.225.2) and the publicGET /api/config, and prior art that uses the same gateway for exec, ControlMaster, and git over SSH.psor attachments, create with size and labels, file/git in the sandbox, include file mode 0600 and no token in persisted config, sub-agent fork into the same sandbox, out-of-band stop then next turn restarts, keepalive and orphan stop, archive stop, dirty delete refused, clean delete removes the sandbox, existing-sandbox attach, concurrent and failure shapes (stopping, starting, GET 500, hung GET, out-of-band delete and token revoke), and phone-width (375 and 390 px) and 1440 px screenshots. Every round ended with 0 leftover sandboxes and 0 live tokens beyond the deliberate orphan fixture.xum.instancelabel throughPUT /sandbox/{id}/labelsand no sweep stops the handed-off sandbox; a failed relabel refuses a normal removal; releasing access stops the ControlMaster and forward; forced removal on a stopped sandbox names the checkout left on it; missing and Windows-default snapshots refused before any create; 0 live tokens at the end). Its one issue, a forced removal hiding a failed relabel, is fixed in bf08d89.rejects, a dedupe the fake control plane hid) were fixed.tests/runtime/daytonaRuntime.test.tsruns against real Daytona only withTEST_DAYTONA=1; it is not in the unit suite.Risks
SSHRuntime.ts,SSH2ConnectionPool.ts,sshConnectionPool.ts,sshConfigParser.ts(Includesupport),muxSshConfigWriter.ts, and the OpenSSH/SSH2 transports gained hooks for host-key pinning, token refresh, and refused-auth recovery. Plain SSH and Coder workspaces go through these paths. The new host-key, redaction, and token hooks act only on Daytona aliases; the delete safety check moved into an overridablecheckWorkspaceDeletablewith the same script. Two changes apply to every host: the ssh2 config parser now followsInclude(up to 16 levels; an Include under aHostblock applies only when that block matches, underMatchonlyMatch allexpands, and the includer's scope is restored after it), and writing Xum's managed blocks (Coder and Daytona) into~/.ssh/confignow also takes a cross-process lock file next to the resolved config.-Rforwards for the AI proxy,-A) is untested until a valid key is available. Everything is behind an explicit runtime choice.Follow-ups
runtime_start_failedalthough its bash runs after the restart (startup-file load runs before the runtime restarts the sandbox).self -> .symlink loop inside a skill is copied until the seed's depth limit (8 levels).._-+@,=~and path separators), with an error that names it.~/...).xum server) that remove the last two workspaces of one owned sandbox at the same time both keep it; the orphan sweep stops it but nothing deletes it. Deleting needs the in-memory "removal passed its checks" record persisted where both backends can read it.