Repository navigation
Drain native self-host requests before stopping workerd #109
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
1f2f102
ae64459
df3c361
f520584
df9f70f
e3ac794
bc5ea05
3502bda
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -42,12 +42,15 @@ session and encryption keys in the persistent volume on first boot. | |
|
|
||
| ```sh | ||
| docker pull ghcr.io/usefulsoftwareco/executor-selfhost:beta | ||
| docker run --detach --name executor-v2 --init --restart unless-stopped \ | ||
| docker run --detach --name executor-v2 --init --restart unless-stopped --stop-timeout 30 \ | ||
| --publish 127.0.0.1:4400:4400 \ | ||
| --volume executor-v2-data:/app/data \ | ||
| ghcr.io/usefulsoftwareco/executor-selfhost:beta | ||
| ``` | ||
|
|
||
| Docker's default 10-second stop timeout is shorter than the host's request drain | ||
| plus child process grace periods; keep `--stop-timeout 30` when recreating the container. | ||
|
|
||
| Open `http://localhost:4400` and create the first administrator. The container | ||
| includes the dashboard, API, MCP server, database, and app runtime. For a pinned | ||
| release, use a published version such as `2.0.0-beta.N` instead of the moving | ||
|
|
@@ -263,6 +266,8 @@ Create an image service from `ghcr.io/usefulsoftwareco/executor-selfhost:beta`, | |
| attach a new volume at `/app/data`, and generate a public domain routed to port | ||
| `8080` (Railway injects `PORT=8080`; use your own value if you override it). | ||
| Set the healthcheck path to `/health` with a 120-second startup timeout. | ||
| Set `RAILWAY_DEPLOYMENT_DRAINING_SECONDS=30` (or `deploy.drainingSeconds` to `30`) | ||
| so Railway gives the host time to drain before SIGKILL; the default is zero. | ||
| Deploy, or redeploy if the service already started, so it picks up the new | ||
| public domain. Open the domain and create the first administrator. | ||
|
|
||
|
|
@@ -399,6 +404,16 @@ To stop the server while retaining its data: | |
| docker compose -f apps/hosted/self-host/compose.yaml down | ||
| ``` | ||
|
|
||
| On SIGTERM or SIGINT the native host stops accepting new requests and gives | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. LOW 1: a terminal Ctrl-C skips this drain.
Fix: command.SysProcAttr = &syscall.SysProcAttr{Pdeathsig: syscall.SIGKILL, Setpgid: true}With that change, the same test returned 200 after 3.0 s and exited 0, and SIGKILL on the host still left no orphans.
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in bc5ea05. Linux children now use their own process groups while retaining Pdeathsig; macOS is unchanged. The new test sends SIGINT to an isolated host group and verifies its child survives to exit cleanly on SIGTERM. |
||
| in-flight work up to 10 seconds to finish. Open GET event streams and NDJSON | ||
| responses end immediately with a clean EOF, even if a finite NDJSON response | ||
| has more records; upgraded connections also close immediately. A second signal | ||
| ends the request drain without shortening the child processes' grace periods. The | ||
| product and apps get 10 seconds together to exit, then the telemetry collector | ||
| gets 3 seconds. If a child has to be killed, the host names it in the log and | ||
| exits nonzero. | ||
| On Linux, terminal Ctrl-C drains the host just like `docker stop`. | ||
|
|
||
| ## App catalog | ||
|
|
||
| Onboarding and **Add app** list Executor's official apps from the published | ||
|
|
@@ -577,10 +592,12 @@ the body reaches a sleep before each wake. | |
|
|
||
| The product process has its own watchdog. Once the product has answered its first health check, the | ||
| host probes `/health` every `EXECUTOR_WATCHDOG_INTERVAL` seconds (default 10). After | ||
| `EXECUTOR_WATCHDOG_FAILURES` failed probes in a row (default 9, about 90 seconds), it kills workerd | ||
| and exits non-zero. Run the container with a restart policy such as `restart: unless-stopped` (the | ||
| bundled compose file sets it), so a fresh process starts. `EXECUTOR_WATCHDOG=off` turns both | ||
| watchdogs off. | ||
| `EXECUTOR_WATCHDOG_FAILURES` failed probes in a row (default 9, about 90 seconds), it logs | ||
| `Executor watchdog: the product did not answer …` and cuts public requests without draining. | ||
| It then sends SIGTERM to workerd, gives the product and apps five seconds together to stop, | ||
| kills any that are still running, and exits non-zero. Run the container with a restart policy such | ||
| as `restart: unless-stopped` (the bundled compose file sets it), so a fresh process starts. | ||
| `EXECUTOR_WATCHDOG=off` turns both watchdogs off. | ||
|
|
||
| `EXECUTOR_APP_PROCESS=shared` runs app code in the product's process as earlier releases did; then | ||
| only the product watchdog applies, and a looping app restarts the whole host. Exports always use | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LOW 2: Railway gets no drain by default.
Railway's
RAILWAY_DEPLOYMENT_DRAINING_SECONDSdefaults to 0: per Railway's docs, the deployment "is given 0 seconds to gracefully shutdown before being forcefully stopped with a SIGKILL". So on Railway the new drain never runs, and Pdeathsig kills the children mid-request.Fix: next to this
--stop-timeout 30note, and in the Railway section (## Deploy on Railway, and## Railwayinapps/hosted/self-host/README.md), say to setRAILWAY_DEPLOYMENT_DRAINING_SECONDS=30(ordeploy.drainingSeconds).There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in 3502bda. The Docker timeout note and both Railway sections now document setting RAILWAY_DEPLOYMENT_DRAINING_SECONDS=30 (or deploy.drainingSeconds to 30). No Railway config files were added.