[CMGS]

Closes #475. Adds a persistent per-node cordon next to the transient drain, documents the admission model as one table, and renames the SDK verbs so each one names a single hold.

Answers to #475

1. One admission table: docs/cluster.md#node-admission, with five states (serving, cordoned, draining, draining with evacuate=1, cordoned and draining) against the six questions. sandboxd-api.md links to it instead of repeating it.
2. States: cordoned is new.
   • It is persisted as <data_dir>/cordoned.
   • New claims, forks and checkpoint claims answer 429 node cordoned.
   • Existing claims keep working.
   • A root live move in is admitted under the caps and move_concurrency.
   • The node adopts no other node's archived claims, does not refill, and gossips no pools.
   • draining keeps its meaning: transient, refuses moves in. The two states are independent, and each DELETE lifts only its own.
3. Caps: no config change. 0 or omitted stays "no cap"; to take nothing, cordon the node. Changing what 0 means would touch every deployed config. The table and the max_claims row in deploy.md say so. Open to overrule.
4. SDK verbs: Go and Python Drain, Undrain, Cordon, Uncordon. See the decision below.
5. Adoption and caps: answered in the companion PR (test only). A serving node adopts regardless of max_claims, which is documented in cluster.md. max_running binds at the wake. A cordoned or draining node never adopts.

SDK verb decision (for CMGS)

Uncordon (Go) and uncordon() (Python) used to send DELETE /v1/drain. They now send DELETE /v1/cordon; the new Undrain / undrain() sends DELETE /v1/drain.

#475 asks for renaming "without breaking callers", and this is a behaviour change for an existing call. I chose it because it fails closed:

• An old caller that pairs Drain with Uncordon leaves the node drained. The response and Info report draining: true.
• The only alternative that keeps old callers working is Uncordon lifting both holds. That would silently stop a running evacuation whenever someone lifts a cordon.
• No SDK caller outside this repo was found in the workspace.

Release note text:

SDK: Uncordon now lifts the persistent cordon (DELETE /v1/cordon). To lift a drain, call the new Undrain. Code that pairs Drain with Uncordon leaves the node drained until it calls Undrain.

Design

• State: Manager.cordoned sits next to draining, both under m.mu.
  • Cordon writes the marker with utils.WriteFileSync, then sets the flag. Uncordon removes the marker and syncs the dir, then clears the flag.
  • cordonMu orders the two writers.
  • NewManager stats the file before Run, so a node that boots cordoned never starts refill or a golden build.
• Check sites:
  • quotaErr refuses on the cordon and delegates to capacityErr. That covers claim, fork, checkpoint claim, the fork re-check and WarmCounts gossip.
  • admitMoveLocked calls capacityErr; CommitMove still checks only draining.
  • Refill parks on closedLocked() (draining or cordoned).
  • adoptArchived refuses when cordoned, after the held case, so the node keeps re-adopting its own rows.
  • Wake, hibernate, renew, exec, env and release are untouched.
• Hot path: one extra bool read under the m.mu that quotaErr, WarmCounts and refillOnce already hold. No allocation, no new lock, no I/O. File I/O happens only on the two endpoints and once at boot.
• Drain behaviour change: refill now parks a closed node instead of returning early. A pool removed by SetPools while a clone was in flight is therefore dropped on a drained or cordoned node too. Before this, it lingered until the node opened, which under a days-long cordon has no bound.
• Internal rename: pool Manager.Uncordon (which lifted a drain) is now Undrain.
• Metrics: adds sandboxd_cordoned; the sandboxd_draining help text no longer says "cordoned".

Tests

New pool tests:

• TestCordonRefusesClaimsTrimsWarmAndSurvivesARestart
• TestACloneInFlightAtCordonIsNotKept
• TestACordonedNodeStillDropsAPoolRemovedMidClone
• TestCordonRefusesForksAndCheckpointClaims
• TestACordonedNodeKeepsServingItsOwnClaims
• TestACordonedNodeNeverAdoptsButKeepsItsOwnRowAcrossARestart
• TestAMoveLandsOnACordonedNode
• TestCordonOutlivesAReloadAndAnUndrain
• TestCordonIsAudited

Server tests:

• TestCordonEndpoints
• TestCordonPersistFailureAnswers500AndKeepsState
• tenant 403 rows

SDK tests:

• Go TestDrainCordonVerbs
• Python test_node_hold_verbs_hit_their_routes

Mutation controls: each check site was broken one at a time, and a test failed for every one:

• capacityErr in move-in
• loadCordon
• the cordon in quotaErr
• adoption ignoring the cordon
• the cordon checked before held
• refill ignoring the cordon
• refill keeping a late clone
• no warm trim
• no persist
• the early-return refill

Gates

• GOWORK=off make go-lint: 10 of 10 0 issues. (5 modules × linux/darwin), fmt --diff empty.
• asl -forwarder=false ./... clean for sandboxd and sdk/go on linux and darwin.
• go test -race green for sandboxd and sdk/go.
• ruff format --check, ruff check clean; pytest 277 passed.

Hardware

One test host, NUMA node 1. Two sandboxd nodes (a, b) built from 6a7241a, with cocoon v0.7.8 and CH dev acded4b0, a kit Postgres meta-store cell, and an encrypted mesh. Each node ran in its own mount namespace with its root bound at the same path, because live moves need identical root_dir/run_dir strings. Pools: rt:24.04 none/small, warm 2. Excerpt:

control: a cordoned, b serving   claim on a -> 200 {"redirect": [b]}
POST b /v1/cordon                200 cordoned=true warm 0, marker file present
claim on b (no_redirect)         429 node claim quota reached: node cordoned
claim on b (redirect allowed)    200 {"redirect": [a]}
a cordoned too                   claim on a -> 429 node cordoned (no redirect: b gossips no pools)
renew X / hibernate X / wake X   200 / 204 / 204 (wake 88 ms), b still cordoned
move Y a -> b                    200 rounds 3, downtime_ms 227; b holds Y, a 404
POST b /v1/config/reload         200, b still cordoned
restart b (cordoned)             reconcile adopted 2 claims; t+0 .. t+60 s every 5 s:
                                 warm 0, refilling 0, VM count 2 (the two claims), cordoned true
claim on restarted b             429 node cordoned; wake X 204, Y served
DELETE b /v1/cordon              200, marker file gone, warm 2 after 201 ms
audit.jsonl                      {"op":"cordon"} ... {"op":"uncordon"}
negative control: drain only, restart b
                                 t+0 s draining=false, warm 2, VM count 2 (the reported production case)

b's log over the cordoned restart window holds only startup, reconcile and mesh join, with no refill or golden build. The host was restored to its baseline afterwards (no kit processes, netns, slice or files left).