[CMGS]

The host side of the advanced network mode (#471). Stacked on #474 (guest side: silkd tun0, the packet protocol on vsock 2051); merge #474 first. The proxy path stays the default network mode.

What it does

A pool with "tun": true gives each claim an overlay address and a packet door on vsock port 2051 next to its proxy doors. sandboxd carries the guest's IP packets over VXLAN, on an ordinary UDP socket, to an exit: plain Linux configuration (a learning VXLAN device, forwarding, NAT) on the same host or on a gateway machine. sandboxd needs no TUN, no routes and no NAT rules on the node.

• Identity and policy stay in sandboxd. The source becomes the claim's overlay address whatever the guest wrote; the first packet of a flow is judged by EvalFlow (bare host: "*" rules) and the internal-address guard, which also refuses the overlay subnet and 100.64.0.0/30. Per-claim byte, packet and new-flow rates, a flow cap and a quota apply before any further work; a GSO frame is charged for every packet and header it puts on the wire.
• ARP is sandboxd's. It answers ARP from the exits for its own claims only, announces a claim with a gratuitous ARP on arm (claim, wake, move-in), after an exit reload and every 45 s, and never forwards a frame the guest built. VXLAN and ARP are accepted only from the configured exit VTEPs.
• Addresses. A single node allocates in memory (no disk write on the claim path) and logs releases to tun-freed.log for the reuse window; a mesh requires meta_store.cell and allocates with one statement per claim in a cell table, failing closed while the meta store is down. Addresses follow the claim through hibernate, archive, checkpoint, fork and live move.
• Switch and label. PUT /v1/sandboxes/{id}/tun (root only) disables or re-enables a claim and sets its exit label, including on hibernated and archived claims. A disable revokes the packet path at once and runs beside any transition; an enable or a label change waits for it.
• Lifecycle. tun_up is sent in the background after arm, so the claim path adds no RPC; a guest of a tun pool that ends up without a door is sent tun_down. A TCP stream that predates an arm (move, wake, restart) resumes: a non-SYN segment reopens its flow under the claim's policy and limits.
• Audit, usage, metrics. Each ended flow writes a tun_flow audit record; with egress_usage_bytes each allowed flow appends an egress_bytes usage event (IP bytes, ip:port reference, upstream tun/<label>). A clean stop ends live flows so they are recorded up to the stop. /metrics adds frame, datagram, ARP and per-reason drop counters.
• Docs. docs/advanced-network.md host section with a single-host exit reference config and a configuration table; deploy.md, sandboxd-api.md, egress.md.

Fuzz-found checksum-offset policy bypass

• What it was. A guest controls the virtio-net header. With VIRTIO_NET_HDR_F_NEEDS_CSUM set, the incremental source rewrite patches the 16-bit word at csum_start + csum_offset. A guest could aim that offset into the IPv4 header, so the rewrite edited the destination address after the policy check had judged the original one: the packet left for a destination the policy never saw. A mismatched GSO type (TCP GSO on a UDP packet) also panicked the segmenter.
• Fix. Parse refuses a frame whose header does not fit its packet: the GSO type must match the protocol, and a partial checksum must start at the L4 header (csum_start 20) with the offset of that protocol's checksum field (16 for TCP, 6 for UDP); ICMP may not ask for one. Such frames drop as gso.
• Test. FuzzParseAndRewrite asserts that after the rewrite the source is the claim address and the destination is unchanged; its corpus seed for the bypass (testdata/fuzz/FuzzParseAndRewrite/a7fe022f…) and the segmenter panic seed (FuzzSegment/3560257e…) run on every go test; TestParseReadsPortsAndRefusesWhatV1DoesNotCarry covers the refused headers. CI runs each of the four fuzz targets for 30 s.

Configuration

The VXLAN port is a config item: tun.port (default 4789) is the exits' port, and tun.listen the node's socket. The CI kernel-exit test uses 4789 inside its own namespaces; the hardware kit used 4790 because the testbed already runs another VXLAN on 4789. Full table in docs/advanced-network.md#configuration; restart-only fields are refused on reload.

Hot-path cost

• Claims of pools without tun: one map lookup.
• Tun claim path: an in-memory address (or one meta-store round trip), a socket bind and one gratuitous ARP; tun_up runs in the background.
• Per packet, user space (go test -bench . ./tunnet/, arm64 laptop, 0 allocs/op each): a 63 KB GSO frame segmented with full checksums and VXLAN headers straight into the send buffer 8.8–9.0 µs (about 200 ns per wire packet); a small packet 104 ns; an inbound packet delivered 251–261 ns.
• Allocations: an alloc profile of the kernel-exit test shows none left on either direction's loop (recvmmsg into owned buffers replaced x/net ReadBatch, which allocated an address per datagram).
• Syscalls, measured on hardware (4 GiB TCP each way, strace -c on sandboxd and the frame/datagram counters):

	upload (guest → exit)	download (exit → guest)
guest frames per GiB	17.5k (about 61 KB each: guest TSO works)	105k to the guest (inbound merging: 7.3 datagrams per frame)
wire datagrams per GiB	768k (44 per frame)	768k
send syscalls	one sendmsg with UDP GSO per frame (about 27 datagrams per call)	one sendto per guest ACK frame
receive syscalls	recvmmsg, 4.3 ACKs per call	recvmmsg, about 30 datagrams per call
vsock syscalls	one read per ~3 frames (256 KiB reader)	one writev per ~6 frames

Hardware

Two bare-metal testbed hosts (call them test1 and test2), a sandboxd on each in one mesh, PostgreSQL meta store, an exit in its own network namespace on each host. A claim on test2 leaves either through the test2-local exit or through the test1 exit (the "remote" exit below).

Functional (all pass). Rows marked † ran in the first hardware round, on this branch before the review fixes; the others ran again on the review-fixed build.

Check	Result
TCP and UDP through the local and the remote exit	pass; UDP RTT p50 87–111 µs local, 132 µs remote
A claim of a pool without tun †	no address, no door, no tun0
8 concurrent claims across both nodes †	distinct addresses
Mesh config without meta_store.cell †	refuses to load
Forged source address from the guest †	leaves as the claim's overlay address (exit conntrack)
Packets to another claim's overlay address †	dropped by sandboxd (policy), exit forward counter 0
Disable a running claim	tun0 removed, PUT 9–20 ms, traffic stops; re-enable restores it within 1 s
Disabled state across restart and hibernate/wake †	kept
Profile switch to the remote exit and back	pass
Live move test1 → test2 with a 4 GiB download and a UDP probe open	downtime 238 ms, download completed, UDP max gap 247 ms
Live move test2 → test1, same	downtime 200 ms, download completed, UDP max gap 168 ms
A download stalled by a move before the resume fix	resumed and completed after the fix was deployed
Hibernate/wake of an enabled claim †	traffic back within 93 ms
sender drops (VXLAN from a non-exit) during all of the above	0

Throughput, interleaved arms in one run (order tun, proxy, proxy, tun), local exit, 4 GiB each way

CH build	path	up Gbit/s	down Gbit/s	sandboxd CPU s/GiB up / down
dev acded4b0	tun	2.61, 2.57	2.10, 2.20	4.5 / 3.5–3.7
dev acded4b0	proxy 3128	4.22, 5.68	2.18, 2.10	0.5–0.6 / 0.18
test build with a rewritten vsock backend	tun	2.57, 2.58	2.50, 2.50	4.5 / 4.5
test build with a rewritten vsock backend	proxy 3128	4.08, 4.94	4.35, 4.01	0.5–0.6 / 0.17

• Before this PR's receive-side work (inbound merging, recvmmsg), tun download was 1.49 Gbit/s and UDP 64-byte delivery 107k pps; it is now 2.10–2.20 and 320–330k pps.
• With the local exit, sandboxd's CPU includes the exit's kernel work: veth runs the exit's decapsulation, NAT and the receiver's TCP stack inside sandboxd's sendmsg (106 µs per 64 KB frame locally, 36 µs to the remote exit). Through the remote exit, upload costs sandboxd 1.9 s/GiB (about 0.6 core at 2.6 Gbit/s).
• With the faster vsock backend the proxy's download doubles while tun's moves from 2.1 to 2.5 Gbit/s: the tun path is now limited on the host side. Upload stays at 2.6 Gbit/s for every CH build and both exits, with sandboxd at 0.6 core through the remote exit and with a guest send buffer raised to 16 MiB (2.77–2.84 Gbit/s, queue full), so the upload limit is a rate limit in the per-stream pipeline, not CPU and not the TCP window. Remote-exit download is 1.47–1.73 Gbit/s at 3.9–4.0 s/GiB: recvmmsg returns about 15 datagrams per call there instead of 30, so fewer segments merge (5.2 per guest frame) and the guest ACKs every ~10 KB. These are tracked as follow-ups (below), not changed here.

Coverage

Flow / window	How covered
Claim, release, fork, checkpoint, promote refusal, hibernate/wake, disable/enable, PUT on a hibernated/archived claim, archive-row resend	unit tests (pool/tun_test.go), hardware for claim, release, hibernate/wake, disable/enable
Live move with the claim's exit on one of the two move ends	hardware (both directions, TCP + UDP open)
Remote exit (exit on the other sandboxd node)	hardware: traffic, profile switch, throughput
Remote exit distinct from both the box's node and both move ends	not tested on hardware. The exit host in every hardware run is one of the two sandboxd nodes; no third host was available. Covered by: TestTrafficCrossesAKernelExit (a real Linux exit in its own namespaces: learning VXLAN, arp_accept, masquerade, forward and input drops, run as root in CI), TestRouterAnnouncesRewritesAndPassesOnlyReplies and TestRouterAnswersARPOnlyForArmedClaims (gratuitous ARP to the exit picked by label, ARP answered only for armed claims and only from exits), TestRouterResumesAStreamOpenedBeforeTheArmButRefusesAnOpener (a stream continues on a new arm), the pool move/adopt tests that carry address, switch, label and quota in the claim row. What none of these exercise: an exit's FDB moving between two nodes that are both remote to it, under real traffic.
Mesh address allocation	PostgreSQL tests against a real server (uniqueness across nodes, idempotency, reuse window, cells, fail closed); hardware PG
Untrusted guest input	fuzz targets (frame length, parse and rewrite, segmenter, VXLAN and ARP) plus unit tests for each drop reason
Crash windows (assign → commit, commit → release)	read only; residue listed under known gaps

Review

A full-file correctness and flow review, a second flow review on the final diff, a style review and a security review ran on this branch; every finding was checked against the source. Applied, each with a regression test that fails when its fix is reverted:

• one oversized guest packet looped the UDP sender forever; packets and segments over the stream's MTU, and GSO frames over 128 packets, now drop as size
• a tiny gso_size multiplied wire traffic past the rate limits and the quota; frames are charged per wire packet
• the flow table had no default cap (now 16384 flows, 1000 new flows/s); rate buckets reset on reconnect (limits now live with the claim)
• VXLAN and ARP were accepted from any sender
• ICMP errors quoted the overlay address in both directions, so path MTU discovery and port-unreachable never reached the right socket
• a disable racing a wake could leave the door on; a stale tun_up goroutine sent tun_down after a re-arm
• a resend after a failed archive-row write was a no-op
• the quota counter was lost on a clean restart and on a move; live flows were never recorded across a clean stop (usage)
• a receive-only TCP stream hung after a move, a wake or a restart (found on hardware)
• tun flows emitted no egress_bytes usage event
• tun_up and an fsync of the whole release map ran on the claim path; the PG allocator took three round trips
• inbound bytes counted the virtio-net header; inbound merging ignored the guest's offloads and the TCP window

Not adopted, with reasons:

• PG address rows leak when a release fails while PostgreSQL is down, on a crash between the claims commit and the release, or when a node is lost with running claims. Each case holds one address; with the recommended /8 (16M addresses) this is bounded for years of churn, and a collector needs an owner column and a cross-node sweep. Kept as a known gap.
• Local allocator crash window: a crash in the millisecond between the release commit and the release log entry lets that address be reused inside its window; sandboxd's flow table still keeps the new claim from receiving the old flows.
• Quota counter after a crash or an adoption from a replica row regresses to its value at the last disarm, move or clean stop (documented).
• Cached flow verdicts survive a policy reload until the flow idles out, the same as live CONNECT tunnels on the proxy path.
• Inbound queue memory: up to 1024 queued 64 KiB buffers per stream; a guest that stops reading is closed after 10 s.
• Two nodes answering ARP during a move, and a source crash right after the target's commit: at most 45 s of inbound loss until the next announcement, only if a refresh lands in that window.

Codex review, two rounds on one thread:

• Round 1 reported five findings. Four are fixed, each with a regression test where one is deterministic:
  • a profile change that a concurrent disable overtook re-enabled the claim; the commit now checks the state it planned from and re-plans;
  • checkpoint and promote read the switch without the manager lock;
  • a sweep could record a flow before a packet's bytes were counted;
  • a claim waking into a pool whose tun was turned off kept a stale tun0.
• One was kept with a code argument that Codex accepted: a running claim's replica row picks a switch change up at its next replica capture, the same contract as an env change. The API doc now says so.
• Round 2: "No blocking findings. Round 2 closes the remaining blockers at e7854b7c."

Known gaps and follow-ups

• Throughput: the per-stream upload limit (2.6 Gbit/s), per-ACK sendto on download (batching small frames into sendmmsg), and fewer merged segments through a remote exit. Tracked in tun: host packet loop throughput leads after #478 #479 with the profiles above.
• IPv4 only; one stream per guest; Windows guests have no silkd.
• Android guests: redroid's TCP goes through redsocks to the proxy; UDP and ICMP use tun0. Switching redroid to tun0 is cocoonbox's change.

Gates

GOWORK=off make go-test (every module, race plus one pass of every benchmark), make go-lint (5 modules × linux/darwin, 10× 0 issues), asl ./... on both GOOS, go test -race ./... on macOS and in a Linux container, the kernel-exit test as root in a Linux container, 30 s per fuzz target, the PostgreSQL tests against a real server.