Skip to content

silkd: accept vsock connections close-on-exec atomically - #540

Merged
CMGS merged 2 commits into
mainfrom
silkd/accept-cloexec
Nov 9, 2026
Merged

CMGS merged 2 commits into
mainfrom
silkd/accept-cloexec

Conversation

@CMGS

@CMGS CMGS commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

vsock 0.5.4 calls accept(2) and only afterwards sets FD_CLOEXEC with fcntl. An exec spawned on another tokio worker during that window inherits the new silkd control connection, and the child can read that client's requests and write frames to the host.

Change

  • sysutil::vsock_accept calls accept4(SOCK_CLOEXEC | SOCK_NONBLOCK) and returns the fd together with the peer cid.
  • vsock::serve binds the listener through vsock, which already creates it with SOCK_CLOEXEC, and registers it with AsyncFd. It accepts through async_io, so WouldBlock and readiness are handled by tokio.
  • The host-only cid check and the transient-error backoff are unchanged.

Evidence (hardware, base e38c762)

  • Root execs on a 4-vCPU guest: 0.1–0.43% inherited a live root control connection (local port 2048, peer the host).
  • 1-vCPU guest: 0 of 6100. It runs a single runtime thread, so the race window never opens.
  • Named-user execs: 0 of 9000.
  • Fix verified on hardware: pending. The fdrace run on a guest image with this silkd will be posted here.

Gates (Linux arm64 container, rust 1.99.1)

  • cargo fmt --check: clean.
  • cargo clippy --all-targets -- -D warnings: clean.
  • cargo test: all pass.

vsock 0.5 sets FD_CLOEXEC with fcntl after accept(2), so an exec spawned
on another runtime thread in that window inherits the new control
connection. On a 4-vCPU guest 0.1-0.43% of root execs held another live
silkd connection; accept4 with SOCK_CLOEXEC closes the window.
@CMGS

CMGS commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Hardware verification: pass

Setup. Two identical 4-vCPU pools with tun on, one claim per arm. Both guest images are stock rt:24.04 (sha256:2cc9c5a622f8) with only /usr/local/bin/silkd replaced. Inside each guest, the sha256 of /proc/$(pidof silkd)/exe matched its arm.

  • A: 707ef2c (sha 7b746ce11d02), byte-identical to the stock silkd.
  • B: b16d18d (sha 4af5de047cc7). Its diff against A touches silkd/src/{sysutil,vsock}.rs only.
  • Rest of the stack: sandboxd 707ef2c, cocoon v0.7.8, CH 750aa2b.

Method. Each pass ran N execs, each over a fresh vsock connection. Every child listed its own socket fds and reported SO_DOMAIN, getsockname and getpeername for each.

Leaks (leaked sockets / execs; passes in run order, ABBA):

Execs A (707ef2c) B (b16d18d)
root, 3000 @64, ABBA ×4 9, 5, 6, 11, 2, 9, 2, 14 0 in all 8 passes
root, 6000 @128, ABBA 26, 17 0, 0
root total 101 / 36000 (0.28%) 0 / 36000
named user, 3000 @64, ABBA 0, 0 0, 0
  • Every A leak was AF_VSOCK local cid=3 port=2048 peer cid=2, another client's live silkd control connection.
  • No pass reported an error.

Control path on B, once each, all OK:

  • exec: exit code as scripted; a named-user exec ran as the named user; sandboxd HTTP exec returned 0.
  • pty: 3 of 3 sessions completed with no output lost.
  • Files: a 1 MB fs_write, then fs_read with a matching sha256, then fs_stat.
  • tun: tun_down 14.6 ms and tun_up 1.0 ms, with UDP echo over tun0 20/20 both before and after.

Exec throughput (execs/s, median of the passes): mixed, and not attributable to the change.

Execs A B B vs A
root @64 1165 1018 −12.6%
root @128 1123 1146 +2.0%
named user @64 738 773 +4.7%
  • The two arms ran on different VMs, on a host at load average 164–170.
  • An accept takes microseconds, and this run made about 1k accepts per second, so the accept path cannot account for a 12% gap.
  • A run that swaps the two images between the pools would settle it, if wanted.

vsock_accept checks for a pending signal before it returns EAGAIN, so a
nonblocking accept4 returns EINTR when a SIGCHLD from an exited exec lands
on the accepting thread. tokio-vsock retried it at once; the new loop took
the 50 ms error backoff on every such accept.
@CMGS

CMGS commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up e4d877c. When a signal is pending, vsock_accept returns -EINTR even for a non-blocking accept4: it checks signal_pending(current) before timeout == 0. tokio's SIGCHLD reaper triggers this on exec-heavy guests. tokio-vsock retried EINTR at once, but this PR's loop took the 50 ms error backoff. Interrupted now retries at once. This may account for the −12.6% root @64 throughput gap in the hardware comment above, and a rerun on this commit will confirm.

@CMGS

CMGS commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Hardware rerun on e4d877c (EINTR retry): pass

Same host and stack as the earlier runs. The two arms were swapped between two 4-vCPU pools in both orders, ABBA ×4 per order, 3000 root execs @64 per pass. In-guest silkd sha matched each arm.

Execs A 707ef2c C e4d877c
root @64, n=16, median (range) 1238 (1137–1427) 1169 (1126–1433)
root @128, one pass per arm per order 1150 / 1305 1288 / 1259
leaks, root 164 / 60000 0 / 60000
  • C vs A @64 is −5.6% at the median, Mann-Whitney p=0.20, with overlapping ranges.
  • The sign flips by pool: C is faster in pool 1 (1332 vs 1240) and slower in pool 2 (1145 vs 1233). So the difference is noise on a host at load average 154–166.
  • The −29% measured on b16d18d is gone: C's minimum, 1126, is above b16d18d's maximum, 937. It was the 50 ms backoff on EINTR.
  • No guest journal shows a silkd: accept: line.

@CMGS
CMGS merged commit 763fe16 into main Nov 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant