Secretli server

The API behind Secretli. It stores secrets that were encrypted before they arrived, hands them back to whoever proves they hold the link, and forgets them when they are opened or expire.

The server cannot read what it stores. Keys are derived and used only by the clients, the web app and the command-line client, which follow the format specified in secretli/format. The link's secret travels in the URL fragment, which never reaches a server. What arrives here is ciphertext, an encrypted metadata envelope, and three tokens, of which only SHA-256 hashes are kept. CI checks that the format library, the code that could derive a key, is never among this server's dependencies.

What it does

• Uploads: a secret arrives as a multipart upload session. Its parts are streamed into S3-compatible storage and assembled when the session completes, so no request ever carries the whole secret. Part n starts at (n−1) × 32 MiB and is exactly 32 MiB long, but for the last, which holds the rest; any other part is refused, so a session never stores more than it declared. A session that sends no new part for 30 minutes is abandoned, its reservation and its public id freed; the absolute limit stays 24 hours. Each client address may declare up to UPLOAD_BUDGET_BYTES_PER_HOUR for new uploads per hour, counted in memory only; past it, a start gets 429 with Retry-After. The secret's lifetime counts from the completed upload. Its times are kept to the minute, with one minute added so that it never lives shorter than chosen, and none of them tells how long the upload took.
• Retrieval: the link's metadata token unlocks the encrypted metadata. Its blob token opens a 15-minute retrieval session that reads the bundle by byte range. Opening a one-time secret closes it: nobody can open it again, and only the download that opened it can still read it, until that session ends. It closes only once storage shows it holds the object, so a one-time secret opened while storage is down stays live. A wrong blob token, a wrong password as a rule, gets 403 with details.attempts_left; the tenth deletes the secret as its owner could, so a password can't be guessed at for the secret's whole lifetime.
• Owner status: a link can tell whether a recipient has opened a reusable secret. Never when. Once a secret is deleted, expires or, being one-time, has been opened, nothing about it is told: its link gets the same 404 as one that never existed.
• Deletion: the owner link's deletion token removes a secret at once; its object leaves storage within a minute.
• Reserved links: a public id stays taken until the expiry of the secret that took it, also after the secret was opened or deleted, so nobody holding the link can put other content under it (see SECURITY.md).
• Short-code hand-off: a relay through which two devices pass a link after a password-authenticated key exchange. The server only sees public key-exchange shares and ciphertext.
• Cleanup: a worker runs every minute. It forgets secrets past their expiry and opened one-time secrets whose download ended, frees public ids past their expiry, abandons uploads that ran out of time, and is the only code that deletes from storage: every object is on file from before it is written until after it is deleted, so storage never holds one the database has forgotten.
• Storage outages: while object storage can't be reached or answers with server errors, every call that needs it (starting, uploading and completing an upload, opening a one-time secret, reading a range) gets 503 with Retry-After, which the clients wait out and repeat a few times; the cleanup removes in a later cycle whatever it could not remove. A one-time secret whose download an outage interrupts can still be lost once its session ends.
• Logs: a request is logged by its route, not its path, so the logs don't say which secret was asked for.

Endpoints

Method and path	Purpose
POST /api/v1/secrets/uploads	start an upload session for a new secret
PUT /api/v1/secrets/uploads/{session}/parts/{n}	upload one part
POST /api/v1/secrets/uploads/{session}/complete	turn the parts into the secret
DELETE /api/v1/secrets/uploads/{session}	abandon an upload
GET /api/v1/secrets/{id}/meta	the encrypted metadata, and whether a recipient opened a reusable secret
POST /api/v1/secrets/{id}/retrieval-session	open the secret for reading
GET /api/v1/secrets/{id}/blob	read a byte range within a retrieval session
DELETE /api/v1/secrets/{id}	delete, with the owner's deletion token
POST /api/v1/transfers, /claim; POST and GET /{id}/answer, /{id}/delivery; DELETE /{id}	the short-code relay: open and claim a transfer, write each leg once and wait for it up to 25 s, end it early
GET /api/v1/version	the commit the server was built from
GET /api/v1/health/live, /api/v1/health/ready	liveness and readiness
GET /metrics	Prometheus metrics, optionally behind a bearer token

The secret, upload and transfer endpoints are rate limited per client address; health, version and metrics are not.

Running it

The server needs PostgreSQL and an S3-compatible object store. Migrations run at startup.

docker compose -f docker/docker-compose.yml --profile app up -d

starts the server on http://localhost:8080 with PostgreSQL and SeaweedFS. Without --profile app you get only the two backing services, for running the server from source:

docker compose -f docker/docker-compose.yml up -d
cp .env.example .env
make dev

Configuration

Environment variables; see .env.example.

Variable	Description	Default
SERVER_PORT	HTTP port	8080
DATABASE_URL	PostgreSQL connection string	—
S3_ENDPOINT	URL of the S3-compatible API, with http:// or https://	—
S3_BUCKET	bucket name, which must exist	—
S3_ACCESS_KEY / S3_SECRET_KEY	S3 credentials	—
S3_REGION	region for request signing	us-east-1
MAX_FILE_SIZE	encrypted upload limit in bytes	1073741824 (1 GiB)
MAX_STORED_BYTES	cap on what all secrets together may take up in storage; a new upload past it gets 503 with Retry-After (0: no cap)	0
CLEANUP_INTERVAL	how often the cleanup runs	1m
ALLOWED_ORIGINS	CORS origins, only needed when the web app is served from another origin	—
METRICS_TOKEN	bearer token required for /metrics	—
TRUSTED_PROXIES	IPs or CIDRs of reverse proxies whose X-Forwarded-For is trusted for rate limiting	—
RATE_LIMIT_MULTIPLIER	raises every rate limit by this whole factor, for test environments only	1
UPLOAD_BUDGET_BYTES_PER_HOUR	bytes one client address may declare for new uploads per hour and instance, raised by RATE_LIMIT_MULTIPLIER; at least MAX_FILE_SIZE (0: off)	4294967296

Behind a reverse proxy, set TRUSTED_PROXIES to the proxy's address so rate limits key on the real client; otherwise forwarded headers are ignored.

Rate limits apply per client address, and an end-to-end suite sends far more requests from one address than a visitor would. Test environments set RATE_LIMIT_MULTIPLIER (for example to 100) instead of waiting out the limits. It is read once at startup and never from a request, anything but a whole number of at least 1 stops the server from starting, and a server with raised limits logs a warning. Production leaves it unset.

Storage lifecycle

The cleanup deletes every object itself, so the bucket needs no lifecycle rules to stay clean. In production it carries two anyway, as a safety net for what a database restore or a bug could leave behind. They are in deploy/storage-lifecycle.json, both on blobs/, where every secret's object lives:

• objects expire after 10 days. A secret's object lives at most about 8: an upload of up to 24 hours, then the longest lifetime of 7 days;
• unfinished multipart uploads are aborted after 2 days, since an upload session lasts at most 24 hours.

Storage counts from when an object was made and acts once a night, so it removes an object later than that, never earlier.

The rules live in the bucket, not in the code, so they are applied by hand once, and again whenever the file changes:

aws s3api put-bucket-lifecycle-configuration --bucket "$S3_BUCKET" --endpoint-url "$S3_ENDPOINT" \
  --lifecycle-configuration file://deploy/storage-lifecycle.json
aws s3api get-bucket-lifecycle-configuration --bucket "$S3_BUCKET" --endpoint-url "$S3_ENDPOINT"

TestStorageLifecycleOutlastsEverySecret keeps the rules in step with the server: it fails when a longer lifetime or upload session would let them take an object the server still needs.

Metrics

GET /metrics serves Prometheus metrics: Go runtime and process metrics, and these.

Metric	Type	What it counts
secretli_http_requests_total	counter	requests, by method, route and status code
secretli_http_request_duration_seconds	histogram	request durations, by method, route and status code
secretli_http_in_flight_requests	gauge	requests being answered now
secretli_secrets_created_total	counter	secrets created
secretli_objects_deleted_total	counter	objects the cleanup deleted from storage
secretli_cleanup_errors_total	counter	errors of the cleanup worker, database or storage
secretli_secrets_live	gauge	secrets that can be opened now, by kind: one_time or reusable, which add up to the total
secretli_stored_bytes	gauge	bytes the secrets holding an object take up in storage, uploads under way included
secretli_stored_bytes_limit	gauge	MAX_STORED_BYTES, when set
secretli_secrets_overdue	gauge	secrets past their expiry that the cleanup has not forgotten yet; normally 0
secretli_objects_doomed	gauge	objects waiting for the cleanup to delete them from storage; normally near 0
secretli_objects_removal_failed	gauge	doomed objects that storage has refused to delete at least once; normally 0
secretli_uploads_stuck	gauge	uploads still under way past their own expiry, which the cleanup should have abandoned; normally 0
secretli_public_ids_reserved	gauge	links of secrets that are gone, kept reserved until their expiry

The gauges from secretli_secrets_live on are read from the database at every scrape, so every replica reports the same values: across replicas, take one of them with max, not sum. For example, all live secrets are sum(max by (kind) (secretli_secrets_live)). They are totals of the current state: no secret is told apart, and nothing is stored for them. Uploads and downloads under way, one-time secrets whose download is still running and open short-code transfers are not counted on purpose: each lasts only while one person acts, and a count of them, scraped every minute into a monitoring store, would record when someone uploaded, downloaded or handed over a secret.

Suggested alerts:

Alert	PromQL	For
The cleanup is not forgetting expired secrets	max(secretli_secrets_overdue) > 0	10m
Storage keeps refusing to delete objects	max(secretli_objects_removal_failed) > 0	30m
The cleanup is not abandoning expired uploads	max(secretli_uploads_stuck) > 0	30m
Storage is nearly full	max(secretli_stored_bytes) / max(secretli_stored_bytes_limit) > 0.8
The cleanup fails	increase(secretli_cleanup_errors_total[30m]) > 0

Development

make test        # everything, including the Postgres and S3 integration tests (needs Docker)
make test-short  # unit tests only
make api-test    # the API, over HTTP, against a running server; see below
make lint
make vuln

The API test (apitest/) checks a running server through its HTTP API alone, with no client and no format library: uploads in one and several parts, the upload rules, metadata, retrieval and byte ranges, one-time and reusable secrets, that a gone secret answers like one that never was and its link cannot be reused, deletion, times kept to the minute, and the short-code relay. The server never decrypts anything, so random bytes stand in for ciphertext. It sends more requests than the rate limits allow from one address, so the server under test needs raised limits:

make build && RATE_LIMIT_MULTIPLIER=100 ./bin/secretli   # with DATABASE_URL and S3_* set
SECRETLI_SERVER=http://localhost:8080 make api-test

One test waits about six minutes for a secret to expire and runs only with SECRETLI_SLOW_TESTS=1 set.

CI runs the API test against a fresh server at every change. It also runs the whole of Secretli with the change, from secretli/e2e: the web app and both clients against this server behind a gateway like production's. Only the latest command-line client is supported, so it runs with that one alone. Nothing is published unless both pass.

Database queries are generated with sqlc from internal/adapter/postgres/queries; run sqlc generate after changing them or the migrations.

Images

Every change on main that passes CI is published to the GitHub Container Registry:

ghcr.io/secretli/server:main
ghcr.io/secretli/server:sha-<commit>
ghcr.io/secretli/server:<YYYYMMDD-HHmmss>-<commit>

The images are built for linux/amd64 and linux/arm64 on distroless, carry an SBOM and SLSA provenance, and are signed with cosign, keyless:

cosign verify ghcr.io/secretli/server:main \
  --certificate-identity-regexp 'https://github.com/secretli/server/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Security

See SECURITY.md for reporting vulnerabilities.

License

MIT