[CMGS]

What

POST /v1/drain?evacuate=1 drains a node and moves its claims to peers with their state, so a node can be retired or serviced without waiting for leases to run out. It needs archive_failover.

1. Cordon, as POST /v1/drain does.
2. In the background, on the maintenance budget: hibernate a running claim, archive a hibernated one through the existing archive path.
3. Hand-off: Release(id, expect, owner) moves the row this node owns at epoch expect to no owner at a new epoch, in one compare-and-set. The node then asks a peer to adopt it through GET /v1/sandboxes/{id}/owner with the claim token, and forgets the claim only after a peer answered. With no adopter it takes the row back and keeps the claim.
4. Progress: /v1/info gets evacuating (claims left) and /metrics gets sandboxd_evacuating. Once evacuating is gone, claimed counts the claims that stayed, and the log names each and why.
5. DELETE /v1/drain stops the run. A claim mid-archive finishes its archive and keeps its row.

SDK Close in Go and Python now follows a moved claim: a 404 runs Lookup and releases at the owner it finds.

Fencing and repair

• After a release, every old-owner write is refused. Put is owner-guarded. Replace, Consume and a second Release need the pre-release epoch.
• A release that finds its row moved reads it. A row another node took drops the claim. A row this node still owns at another epoch keeps the claim.
• A row with no owner is one an outage cut short or a later owner released. It may hold a later owner's writes, so the node never takes it back. It asks a peer to adopt, so a repeat evacuate resumes the hand-off.
• A crash between the release and a peer's answer leaves a row no node lists. A restart drops the claim and the next Lookup with its token adopts it. A released row past its retention deadline is purged by the hourly sweep.
• A host-only env write that races a move answers 404, so the client looks the claim up and retries instead of losing the value. One that lands during the archive reaches the row.

Wake gate and queue order

Until its hand-off, a wake of a queued claim answers 503 sandbox moving to another node; retry with Retry-After. The gate sits in beginWake, so an explicit restart of a failed claim still works.

Archived claims move first and running ones last, because a parked claim answers 503 until its turn while a running one keeps serving. The run takes one budget's worth at a time, so crash recovery and reaps of the claims that stay keep their FIFO turn.

These claims stay and keep serving: egress-lane claims, claims with volumes, failed claims, claims whose tenant was removed, claims archived before archive_failover was on, claims whose step failed, and every claim of a node without archive_failover.

Lease across the move

A claim that never opted into archiving keeps its lease deadline. An opted-in claim keeps the later of its lease and archive_delete_after_seconds, or no deadline when that is 0. Either deadline holds while the claim stays archived: its first wake on the adopter starts a fresh lease, like any wake of an archived claim. Sweep archives keep replacing the lease with retention, as before.

Hot path

Nothing on the claim path. A wake of a running claim is unchanged. A wake of a hibernated or archived claim adds one map read under the lock beginWake already takes. Info is O(1). /metrics gains one line.

Measurements

Two nodes, about 11 ms apart, S3 and PostgreSQL on one of them, on the binary of this commit:

42 claims (30 running, 6 hibernated, 6 archived)	FIFO, no batches	this PR
whole run	13.83 s	15.35 s
archived claim answers 404, max	10.96 s	0.28 s
hibernated claim answers 404, max	13.09 s	2.52 s
running claim answers 404, max	not recorded	15.51 s
recovery of a kept claim's dead VMM during a run of 47 claims	15.08 s	6.57 s

A running claim answers 503 for about 1 s of its move. A first wake on the adopter over S3 takes 11-14 s.

Case	Result
mixed running, hibernated, archived	moved in 0.91 s; disk, /dev/shm and a spawned process intact; old owner answers 404; Lookup finds the adopter; Close on a stale handle releases there
12 wakes in flight when the evacuation starts	12 × 204, and one call moved all 12
host-only env writers running through the move	the last acknowledged write is on the adopter for all 12; each writer's first refusal is a 404
uncordon mid-run, 12 claims	9 stayed, 3 moved, none unlisted
SIGTERM and SIGKILL mid-run, 12 claims	not draining after restart; every claim on one node
PostgreSQL stopped mid-run, 12 claims, 5 runs	the rest stay hibernated, skipped in about 0.6 s; wake during the outage answers 204 or 503; repeat evacuates afterwards moved all 12 with none unlisted
a claim with volumes	stays running and serving; plain claims moved
no peer, 6 claims	taken back and archived on the node; wake in about 360 ms
adopter at max_running	adopts archived claims; a wake answers 429 until a slot frees; the cap holds

Not run on hardware: the exact crash window between release and a peer's answer, the egress-lane skip and replicated claims. Those have unit tests only.

Known costs

• With no possible adopter, a run still hibernates and archives each claim before it takes the row back, so streams end. The claims wake normally.
• After a PostgreSQL outage, a peer refuses adoption for one lease grace after its fence lifts. Claims in that window stay on the node, and a repeat evacuate moves them.
• Close of a sandbox that is gone costs one lookup across the peers. A Lookup that cannot reach the adopter also answers nil, in Go, so that claim lives to its deadline, or to its retention once archived.
• The batches add 10-17% to the whole run. A worker pool would avoid that but was not worth its code.

Tests

• Registry contract in archivestest (Mem and PostgreSQL): release, its fencing and take after release.
• Pool: hand-off for each state, refused and lost replies, repeat evacuate resuming, stale wake, the wake gate, queue order, budget fairness, in-flight wake, env write races, removed tenant, lease table.
• store/peer: adoption across peers.
• Server: routing, info and metrics fields, 503 with Retry-After on HTTP and preview.
• SDK: Close on a moved claim and on one no node holds, in Go and Python.