Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .github/workflows/fork-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,21 @@ permissions:
contents: read

jobs:
native:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
- uses: actions/setup-go@v6
with:
go-version-file: apps/hosted/self-host/native/go.mod
cache-dependency-path: apps/hosted/self-host/native/go.sum
- name: Vet and race-test the Go host
working-directory: apps/hosted/self-host/native
run: go vet ./... && go test -race -count=1 ./...

check:
runs-on: ubuntu-latest
timeout-minutes: 30
Expand Down
10 changes: 8 additions & 2 deletions apps/docs/content/run/self-host.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,8 @@ still fail now and then; only three failures in a row count. Give your own probe
1. Attach a new persistent volume at `/app/data`.
2. Generate a public domain in the service's networking settings. Route it to port `8080`, which Railway supplies through `PORT`. If you override `PORT`, use that value.
3. Set the Railway healthcheck path to `/health`, with a startup timeout of 120 seconds.
4. Deploy (or redeploy if the service already started). This loads the new public domain into the server. Open that domain to create the first administrator.
4. Set `RAILWAY_DEPLOYMENT_DRAINING_SECONDS=30` (or `deploy.drainingSeconds` to `30`) so Railway gives the host time to drain before SIGKILL; the default is zero.
5. Deploy (or redeploy if the service already started). This loads the new public domain into the server. Open that domain to create the first administrator.

No database service or secret variables are needed. Executor reads
`RAILWAY_PUBLIC_DOMAIN` and `PORT`. The container prepares the root-owned volume,
Expand All @@ -246,13 +247,18 @@ it. This example uses HTTPS port `4443`, for a host where another service
already uses `443`:

```bash
docker run -d --name executor-v2 --init --restart unless-stopped \
docker run -d --name executor-v2 --init --restart unless-stopped --stop-timeout 30 \
-p 127.0.0.1:4400:4400 -v executor-v2-data:/app/data \
-e BETTER_AUTH_URL=https://<machine>.<tailnet>.ts.net:4443 \
ghcr.io/usefulsoftwareco/executor-selfhost:beta
tailscale serve --bg --https=4443 http://127.0.0.1:4400
```

Docker's default 10-second stop timeout is shorter than the host's request drain
plus child process grace periods; keep `--stop-timeout 30` when recreating the container.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOW 2: Railway gets no drain by default.

Railway's RAILWAY_DEPLOYMENT_DRAINING_SECONDS defaults to 0: per Railway's docs, the deployment "is given 0 seconds to gracefully shutdown before being forcefully stopped with a SIGKILL". So on Railway the new drain never runs, and Pdeathsig kills the children mid-request.

Fix: next to this --stop-timeout 30 note, and in the Railway section (## Deploy on Railway, and ## Railway in apps/hosted/self-host/README.md), say to set RAILWAY_DEPLOYMENT_DRAINING_SECONDS=30 (or deploy.drainingSeconds).

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3502bda. The Docker timeout note and both Railway sections now document setting RAILWAY_DEPLOYMENT_DRAINING_SECONDS=30 (or deploy.drainingSeconds to 30). No Railway config files were added.

On Railway, set `RAILWAY_DEPLOYMENT_DRAINING_SECONDS=30` (or `deploy.drainingSeconds`
to `30`) instead; Railway's default drain is zero seconds.

`tailscale serve status` shows the address. `BETTER_AUTH_URL` must match it
exactly, including a port other than `443`; without the port, browser sign-in
is rejected as an invalid origin. Changing Serve needs administrator rights on
Expand Down
27 changes: 22 additions & 5 deletions apps/hosted/self-host/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,12 +42,15 @@ session and encryption keys in the persistent volume on first boot.

```sh
docker pull ghcr.io/usefulsoftwareco/executor-selfhost:beta
docker run --detach --name executor-v2 --init --restart unless-stopped \
docker run --detach --name executor-v2 --init --restart unless-stopped --stop-timeout 30 \
--publish 127.0.0.1:4400:4400 \
--volume executor-v2-data:/app/data \
ghcr.io/usefulsoftwareco/executor-selfhost:beta
```

Docker's default 10-second stop timeout is shorter than the host's request drain
plus child process grace periods; keep `--stop-timeout 30` when recreating the container.

Open `http://localhost:4400` and create the first administrator. The container
includes the dashboard, API, MCP server, database, and app runtime. For a pinned
release, use a published version such as `2.0.0-beta.N` instead of the moving
Expand Down Expand Up @@ -263,6 +266,8 @@ Create an image service from `ghcr.io/usefulsoftwareco/executor-selfhost:beta`,
attach a new volume at `/app/data`, and generate a public domain routed to port
`8080` (Railway injects `PORT=8080`; use your own value if you override it).
Set the healthcheck path to `/health` with a 120-second startup timeout.
Set `RAILWAY_DEPLOYMENT_DRAINING_SECONDS=30` (or `deploy.drainingSeconds` to `30`)
so Railway gives the host time to drain before SIGKILL; the default is zero.
Deploy, or redeploy if the service already started, so it picks up the new
public domain. Open the domain and create the first administrator.

Expand Down Expand Up @@ -399,6 +404,16 @@ To stop the server while retaining its data:
docker compose -f apps/hosted/self-host/compose.yaml down
```

On SIGTERM or SIGINT the native host stops accepting new requests and gives

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOW 1: a terminal Ctrl-C skips this drain.

configureChild in apps/hosted/self-host/native/process_linux.go:10 sets only Pdeathsig, so the product, apps and collector share the host's process group. A Ctrl-C in a terminal (native dev, or docker run -it) sends SIGINT to every child at once. workerd exits, and in-flight requests are cut with no drain. On this head, a 5 s request got 502 ("Executor did not complete the request") and the host exited in 0.05 s.

Fix:

command.SysProcAttr = &syscall.SysProcAttr{Pdeathsig: syscall.SIGKILL, Setpgid: true}

With that change, the same test returned 200 after 3.0 s and exited 0, and SIGKILL on the host still left no orphans. docker stop, compose and Railway signal PID 1 only, so they're unaffected.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in bc5ea05. Linux children now use their own process groups while retaining Pdeathsig; macOS is unchanged. The new test sends SIGINT to an isolated host group and verifies its child survives to exit cleanly on SIGTERM.

in-flight work up to 10 seconds to finish. Open GET event streams and NDJSON
responses end immediately with a clean EOF, even if a finite NDJSON response
has more records; upgraded connections also close immediately. A second signal
ends the request drain without shortening the child processes' grace periods. The
product and apps get 10 seconds together to exit, then the telemetry collector
gets 3 seconds. If a child has to be killed, the host names it in the log and
exits nonzero.
On Linux, terminal Ctrl-C drains the host just like `docker stop`.

## App catalog

Onboarding and **Add app** list Executor's official apps from the published
Expand Down Expand Up @@ -577,10 +592,12 @@ the body reaches a sleep before each wake.

The product process has its own watchdog. Once the product has answered its first health check, the
host probes `/health` every `EXECUTOR_WATCHDOG_INTERVAL` seconds (default 10). After
`EXECUTOR_WATCHDOG_FAILURES` failed probes in a row (default 9, about 90 seconds), it kills workerd
and exits non-zero. Run the container with a restart policy such as `restart: unless-stopped` (the
bundled compose file sets it), so a fresh process starts. `EXECUTOR_WATCHDOG=off` turns both
watchdogs off.
`EXECUTOR_WATCHDOG_FAILURES` failed probes in a row (default 9, about 90 seconds), it logs
`Executor watchdog: the product did not answer …` and cuts public requests without draining.
It then sends SIGTERM to workerd, gives the product and apps five seconds together to stop,
kills any that are still running, and exits non-zero. Run the container with a restart policy such
as `restart: unless-stopped` (the bundled compose file sets it), so a fresh process starts.
`EXECUTOR_WATCHDOG=off` turns both watchdogs off.

`EXECUTOR_APP_PROCESS=shared` runs app code in the product's process as earlier releases did; then
only the product watchdog applies, and a looping app restarts the whole host. Exports always use
Expand Down
Loading
Loading