You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Self-host server is killed instead of exiting cleanly on SIGTERM #98
On self-host production (ltc-executor, image built from 98d35986), the server does not exit on its own when stopped. The nightly backup (/home/exedev/executor/backup.sh, executor-v2-backup.timer, 09:00 UTC / 01:00 PST) runs docker stop -t 30 executor-v2-server-1:
Nov 17: the stop was logged at 09:00:26Z. About 15 s later (09:00:41Z) the host's reverse proxy logged ReverseProxy read error during body copy: unexpected EOF and Executor request failed: GET read unix @->/tmp/executor-host-…/product.sock: read: connection reset by peer, then Executor: signal: killed. The container restarted at 09:01:13Z and was healthy at 09:01:31Z.
Nov 16: the same pattern (stop at 09:00:36Z, restarted at 09:01:17Z).
Docker reports exit 0, OOMKilled=false. The container runs executor-entrypoint executor-host serve with StopTimeout 30.
So on SIGTERM the Go host (apps/hosted/self-host/native) ends up SIGKILLing the product (workerd) child instead of draining it. In-flight requests are reset rather than finished or refused cleanly.
Why it matters
Every nightly backup and every deploy cuts in-flight requests and kills the product without a clean shutdown. The snapshot is taken after a kill, not after a clean stop.
Rollback rehearsals and cutovers depend on a predictable stop.
Expected
On SIGTERM the host stops accepting new requests, lets in-flight ones finish (or answers 503), asks workerd and the collector to exit, and exits 0 well inside the stop timeout, with no signal: killed.
Notes
Found while tracing an unexplained 01:01 PST restart. It was the backup, not a crash or OOM.
Nov 18 backup check (09:00 UTC / 01:00 PST, image 6fcb0fd0, build 64a2d477)
Result:Not fixed by the upstream sync (#96).workerd was still SIGKILLed after the Go host's 15 s shutdown deadline; the stop took 15.13 s (09:00:26.671Z → 09:00:41.797Z). The Rust telemetry collector stopped cleanly in 1.4 ms, and the container came back healthy after the backup copy.
09:01:02.4406Z: container restarted after the stopped volume copy; ready in 4 s at 09:01:07.135Z, Health: healthy at 09:01:08.843Z, RestartCount: 0, Image: sha256:6fcb0fd08c758e101e693187c05dfe13af9e7642ab6227781631aa9c3f0e9c88.
Root-cause pointer in apps/hosted/self-host/native/main.go
On <-signals (main.go:855-876), serve signals workerd (command.Process.Signal(syscall.SIGTERM) at :868) and waits 15 s (:871-872) before shutting down or closing public (&http.Server{...} at :809-810, which only closes via defer public.Close() when serve returns). Any long-lived streaming GET (e.g. MCP/SSE) through public → product.sock stays open while workerd waits for active connections to finish, exhausting the 15 s timer until command.Process.Kill() fires.
Verified on the first nightly backup with the fix (production build 7193e6a5, image 92662fa7), Nov 20 01:00 PST:
backup.sh stopped the container at 09:00:16.69Z.
The host drained for 10.04 s with one long-lived stream open, then cut it at 09:00:26.73Z (Executor shutdown: requests at cut: map[POST text/event-stream:1]). The collector stopped 0.1 s later (09:00:26.83Z), with the host exiting about 10.15 s after the stop signal.
Log lines in 08:59:30Z–09:03:00Z matching killed, ReverseProxy read error or request failed: 0. Before the fix, every night ended in a 15 s timeout and signal: killed.
The container restarted at 09:01:03Z, was ready in 11 s and healthy at 09:01:21Z, with RestartCount 0. The backup archive was written at 09:05:40Z.
Activity
shuv1337 commentedon Nov 18, 2026
Nov 18 backup check (09:00 UTC / 01:00 PST, image
6fcb0fd0, build64a2d477)Result: Not fixed by the upstream sync (#96).
workerdwas stillSIGKILLed after the Go host's 15 s shutdown deadline; the stop took 15.13 s (09:00:26.671Z→09:00:41.797Z). The Rust telemetry collector stopped cleanly in 1.4 ms, and the container came back healthy after the backup copy.Exact timeline (
ltc-executor)09:00:26.671Z(backup.sh):docker stop -t 30 executor-v2-server-1begins.09:00:26.7039Z→09:00:26.7053Z(docker logs): collector logs{"level":"info","message":"stopping"}then{"level":"info","message":"collector stopped"}(1.4 ms).09:00:41.7534Z:httputil: ReverseProxy read error during body copy: unexpected EOF09:00:41.7537Z:Executor request failed: GET read unix @->/tmp/executor-host-75845027/product.sock: read: connection reset by peer09:00:41.7857Z:Executor: signal: killed09:00:41.7965Z: containerFinishedAt(ExitCode: 0,OOMKilled: false).09:01:02.4406Z: container restarted after the stopped volume copy; ready in 4 s at09:01:07.135Z,Health: healthyat09:01:08.843Z,RestartCount: 0,Image: sha256:6fcb0fd08c758e101e693187c05dfe13af9e7642ab6227781631aa9c3f0e9c88.Root-cause pointer in
apps/hosted/self-host/native/main.goOn
<-signals(main.go:855-876),servesignalsworkerd(command.Process.Signal(syscall.SIGTERM)at:868) and waits 15 s (:871-872) before shutting down or closingpublic(&http.Server{...}at:809-810, which only closes viadefer public.Close()whenservereturns). Any long-lived streamingGET(e.g. MCP/SSE) throughpublic→product.sockstays open whileworkerdwaits for active connections to finish, exhausting the 15 s timer untilcommand.Process.Kill()fires.shuv1337 commentedon Nov 20, 2026
Verified on the first nightly backup with the fix (production build
7193e6a5, image92662fa7), Nov 20 01:00 PST:backup.shstopped the container at 09:00:16.69Z.Executor shutdown: requests at cut: map[POST text/event-stream:1]). The collector stopped 0.1 s later (09:00:26.83Z), with the host exiting about 10.15 s after the stop signal.killed,ReverseProxy read errororrequest failed: 0. Before the fix, every night ended in a 15 s timeout andsignal: killed.RestartCount 0. The backup archive was written at 09:05:40Z.