Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -75,10 +75,20 @@ PORT=7401
# load.
SIDECAR_WORKERS=2

# ----- Build library -----
# SQLite path for persisted trajectories. Empty disables persistence and
# the get_similar_past_builds tool (useful for stateless test deployments).
BUILDLIBRARY_PATH=./buildlibrary.db
# ----- Build library (PostgreSQL) -----
# Connection string for the build library. Required; the API refuses to
# start without it. docker-compose and the Helm bundled Postgres default
# to these credentials; override for managed Postgres.
DATABASE_URL=postgres://robuilder:robuilder@localhost:5432/robuilder?sslmode=disable
POSTGRES_USER=robuilder
POSTGRES_PASSWORD=robuilder
POSTGRES_DB=robuilder

# Lease-based job recovery (multi-pod safe). Defaults shown.
GENERATION_LEASE_TTL=90s
GENERATION_LEASE_SWEEP_INTERVAL=30s
# Requeue cap for jobs whose worker died mid-run. 0 disables retry.
GENERATION_MAX_ATTEMPTS=0

# ----- Generation worker pool -----
# Number of goroutines that pull from the generations queue concurrently.
Expand Down
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ authored for them.
│ │ HTTP handler │ → │ Orchestrator │ │
│ └──────────────┘ │ ↪ LLM provider (Claude / OpenAI) │ │
│ │ ↪ Tool registry │ │
│ │ ↪ Build library (SQLite) │ │
│ │ ↪ Build library (PostgreSQL) │ │
│ │ ↪ Scoring client → ─────┐ │ │
│ └───────────────────────────│──────────┘ │
└─────────────────────────────────────────────────│────────────┘
Expand All @@ -59,7 +59,7 @@ cmd/ one Go binary per directory
verify-mapping/ spot-check tool
internal/
api/ HTTP handlers, OpenAPI types
buildlibrary/ SQLite persistence
buildlibrary/ PostgreSQL persistence
catalog/ //go:embed item/mob/skill data
data/ emulator-source parsers
sources/hercules/
Expand Down Expand Up @@ -103,7 +103,7 @@ What's wired up today:
| Servers | UARO profile shipped; alternates by dropping a YAML under `configs/servers/` |
| Playstyle | PvM tuned; `pvp` / `balanced_*` accepted but less validated |
| LLM provider | Anthropic (default) and OpenAI-compatible (LM Studio, llama.cpp, Ollama, vLLM, ...) |
| Persistence | SQLite at `BUILDLIBRARY_PATH` |
| Persistence | PostgreSQL at `DATABASE_URL` |

---

Expand Down Expand Up @@ -213,8 +213,8 @@ task run:api

```bash
task docker:up # builds both images, runs in the foreground
task docker:down # stops; named volume `ro-builder-buildlibrary` survives
task docker:nuke # stops AND drops the buildlibrary volume
task docker:down # stops; named volume `ro-builder-pgdata` survives
task docker:nuke # stops AND drops the pgdata volume
```

The sidecar's calc backend defaults to rocalc. Set `CALC_BACKEND=stub` in your shell env (or `.env`) before
Expand Down Expand Up @@ -442,8 +442,8 @@ Not yet done, roughly in priority order:
- [ ] Vector search over saved trajectories' reasoning text. Today's `get_similar_past_builds` uses class+scenario
lookup.
- [ ] Add RAG with injecting get_similar_past_builds into initial system prompt
- [ ] Postgres migration for the build library. SQLite pins the API to `replicas: 1` and `strategy: Recreate`; HPA / PDB
are excluded from the Helm chart until this lands.
- [x] Postgres migration for the build library. The API is now stateless; horizontal scaling is supported via the
lease-based generation queue (SKIP LOCKED claim, requeue with cap), configurable replicas, and HPA/PDB in the Helm chart.
- [ ] General prompt tuning - less tokens, more progress.
- [ ] More golden fixtures beyond Taekwon Kid. The shim handles all 45 supported classes already
- [ ] Golden fixtures for Full Support builds
Expand Down
14 changes: 7 additions & 7 deletions Taskfile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@ tasks:
- task: test:sidecar:backend:rocalc

test:sidecar:shim:rocalc:
desc: 'Shim contract tests against rocalc. Proves rocalc satisfies the contract. Needs vendor/.'
desc: 'Shim contract tests against rocalc. Proves rocalc satisfies the contract. Needs vendor/ .'
dir: '{{.SIDECAR_DIR}}'
env:
CALC_BACKEND: rocalc
Expand All @@ -227,7 +227,7 @@ tasks:
- node --test test/shim/*.test.ts

test:sidecar:backend:rocalc:
desc: 'Rocalc-specific tests (numerics, classes). Needs vendor/.'
desc: 'Rocalc-specific tests (numerics, classes). Needs vendor/ .'
dir: '{{.SIDECAR_DIR}}'
env:
CALC_BACKEND: rocalc
Expand Down Expand Up @@ -329,7 +329,7 @@ tasks:
- docker compose down

docker:nuke:
desc: Stop, remove containers AND named volumes (drops the persisted buildlibrary)
desc: Stop, remove containers AND named volumes (drops the postgres data volume)
cmds:
- docker compose down -v

Expand All @@ -340,7 +340,7 @@ tasks:
#
# Day-zero: `task tilt:bootstrap` (one-shot; installs sealed-secrets
# controller, creates namespace), then `task tilt:seal` (encrypts your
# .env's LLM_API_KEY), then `task tilt:up`.
# .env's LLM_API_KEY and Postgres creds), then `task tilt:up`.
#
# The compose path (docker:*) is intact and remains the CI test target.

Expand All @@ -351,7 +351,7 @@ tasks:
- ./scripts/k8s-bootstrap.sh

tilt:seal:
desc: 'Encrypt LLM_API_KEY from .env into deploy/k8s/sealed-secrets/local.sealed-secret.yaml.'
desc: 'Encrypt LLM_API_KEY + Postgres creds from .env into the local SealedSecret.'
cmds:
- ./scripts/seal-secret.sh

Expand All @@ -361,12 +361,12 @@ tasks:
- tilt up

tilt:down:
desc: 'Stop Tilt-managed resources. The PVC (buildlibrary) survives.'
desc: 'Stop Tilt-managed resources. The Postgres PVC survives.'
cmds:
- tilt down

tilt:nuke:
desc: 'Stop Tilt + delete the namespace (drops the PVC and the buildlibrary).'
desc: 'Stop Tilt + delete the namespace (drops the Postgres PVC).'
cmds:
- tilt down
- kubectl delete namespace ro-builder --ignore-not-found
Expand Down
7 changes: 6 additions & 1 deletion Tiltfile
Original file line number Diff line number Diff line change
Expand Up @@ -110,10 +110,15 @@ k8s_yaml(yaml)

_secret_dep = ['sealed-secret'] if _sealed_secret_ready else []

k8s_resource(
'ro-builder-postgres',
labels=['db'],
)

k8s_resource(
'ro-builder-api',
port_forwards=['8080:8080'],
resource_deps=_secret_dep + ['api-binary'],
resource_deps=_secret_dep + ['api-binary', 'ro-builder-postgres'],
labels=['app'],
)

Expand Down
2 changes: 1 addition & 1 deletion cmd/api/healthz.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import (
)

// healthzHandler is a liveness probe target. Returns 200 with a minimal JSON
// body. Intentionally does not touch SQLite or the sidecar; those failure
// body. Intentionally does not touch the database or the sidecar; those failure
// modes belong to richer probes (or to /readyz, when we add one).
func healthzHandler(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
Expand Down
50 changes: 32 additions & 18 deletions cmd/api/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@
//
// Required environment:
//
// BUILDLIBRARY_PATH - SQLite file path for the build library. The DB
// holds the generations queue + saved trajectories.
// Required; the binary refuses to start without it.
// DATABASE_URL - PostgreSQL connection string for the build library
// (e.g. postgres://user:pass@host:5432/robuilder?sslmode=disable).
// Holds the generations queue + saved trajectories.
// Required; the binary refuses to start without it.
//
// Optional environment:
//
Expand All @@ -24,6 +25,9 @@
// GENERATION_POLL_INTERVAL - worker fallback poll cadence (default 5s)
// GENERATION_SHUTDOWN_TIMEOUT - drain window on SIGTERM/SIGINT (default 5m)
// GENERATION_MAX_ITERS - orchestrator tool-use cap (default 60)
// GENERATION_LEASE_TTL - job lease duration (default 90s)
// GENERATION_LEASE_SWEEP_INTERVAL - expired-lease sweep cadence (default 30s)
// GENERATION_MAX_ATTEMPTS - requeue cap; 0 disables retry (default 0)
// LOG_LEVEL - debug|info|warn|error (default info)
// LOG_FORMAT - text|json (default text)
package main
Expand Down Expand Up @@ -64,9 +68,9 @@ func main() {
}

func run() error {
libPath := os.Getenv("BUILDLIBRARY_PATH")
if libPath == "" {
return errors.New("BUILDLIBRARY_PATH is required")
dsn := os.Getenv("DATABASE_URL")
if dsn == "" {
return errors.New("DATABASE_URL is required")
}

logger := buildLogger()
Expand Down Expand Up @@ -104,17 +108,20 @@ func run() error {
}
logger.Info("server profiles loaded", slog.Any("profiles", profileKeys))

lib, err := buildlibrary.Open(libPath)
lib, err := buildlibrary.Open(context.Background(), dsn)
if err != nil {
return fmt.Errorf("buildlibrary open at %s: %w", libPath, err)
return fmt.Errorf("buildlibrary open: %w", err)
}
defer func() { _ = lib.Close() }()
n, err := lib.RecoverOrphans(context.Background())

maxAttempts := envInt("GENERATION_MAX_ATTEMPTS", 0)
rq, fl, err := lib.RecoverExpiredLeases(context.Background(), maxAttempts)
if err != nil {
return fmt.Errorf("recover orphans at startup: %w", err)
return fmt.Errorf("recover expired leases at startup: %w", err)
}
if n > 0 {
logger.Warn("recovered orphaned generations", slog.Int("count", n))
if rq > 0 || fl > 0 {
logger.Warn("recovered expired leases at startup",
slog.Int("requeued", rq), slog.Int("failed", fl))
}

llmCfg := llm.LoadConfigFromEnv()
Expand Down Expand Up @@ -158,12 +165,18 @@ func run() error {
WithCatalog(cat).
WithMaxIters(maxIters)

leaseTTL := envDuration("GENERATION_LEASE_TTL", 90*time.Second)
sweepInterval := envDuration("GENERATION_LEASE_SWEEP_INTERVAL", 30*time.Second)

pool = workers.New(workers.Config{
Library: lib,
Runner: orchestratorRunner{orch: orch},
Save: makeSaveCallback(lib, cat),
Workers: numWorkers,
PollInterval: pollInterval,
Library: lib,
Runner: orchestratorRunner{orch: orch},
Save: makeSaveCallback(lib, cat),
Workers: numWorkers,
PollInterval: pollInterval,
LeaseTTL: leaseTTL,
SweepInterval: sweepInterval,
MaxAttempts: maxAttempts,
})
pool.Start()

Expand Down Expand Up @@ -267,12 +280,13 @@ func (e *apiEnqueuer) ShutdownTimeoutSeconds() int { return int(e.shutdownTimeou
// successful generation to the saved_trajectories table keyed to the
// generation's id.
func makeSaveCallback(lib *buildlibrary.Library, cat *catalog.Catalog) workers.SaveCallback {
return func(ctx context.Context, id string, req orchestrator.GenerateRequest, res *orchestrator.GenerateResult) error {
return func(ctx context.Context, id, owner string, req orchestrator.GenerateRequest, res *orchestrator.GenerateResult) error {
if res == nil || res.Primary == nil {
return errors.New("nil result; nothing to save")
}
in := buildlibrary.SaveInput{
ID: id,
Owner: owner,
Class: req.Class,
Server: req.Server,
Playstyle: req.Playstyle,
Expand Down
47 changes: 47 additions & 0 deletions cmd/api/maintest_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
package main

import (
"context"
"flag"
"fmt"
"os"
"testing"
"time"

"github.com/testcontainers/testcontainers-go"
"github.com/testcontainers/testcontainers-go/modules/postgres"
"github.com/testcontainers/testcontainers-go/wait"
)

// testDSN points at the shared package-level Postgres container. Empty
// under -short (no container started).
var testDSN string

func TestMain(m *testing.M) {
flag.Parse()
if testing.Short() {
os.Exit(m.Run())
}
ctx := context.Background()
ctr, err := postgres.Run(ctx, "pgvector/pgvector:pg17",
postgres.WithDatabase("robuilder_test"),
postgres.WithUsername("test"),
postgres.WithPassword("test"),
testcontainers.WithWaitStrategy(
wait.ForLog("database system is ready to accept connections").
WithOccurrence(2).WithStartupTimeout(60*time.Second)),
)
if err != nil {
fmt.Fprintln(os.Stderr, "testcontainers postgres start:", err)
os.Exit(1)
}
dsn, err := ctr.ConnectionString(ctx, "sslmode=disable")
if err != nil {
fmt.Fprintln(os.Stderr, "connection string:", err)
os.Exit(1)
}
testDSN = dsn
code := m.Run()
_ = ctr.Terminate(ctx)
os.Exit(code)
}
10 changes: 8 additions & 2 deletions cmd/api/readyz_test.go
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
package main

import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"path/filepath"
"testing"
"time"

Expand All @@ -28,11 +28,17 @@ func fakeSidecar(t *testing.T, status int) string {

func openTempLibrary(t *testing.T) *buildlibrary.Library {
t.Helper()
lib, err := buildlibrary.Open(filepath.Join(t.TempDir(), "test.db"))
if testing.Short() {
t.Skip("requires Postgres (testcontainers); skipped under -short")
}
lib, err := buildlibrary.Open(context.Background(), testDSN)
if err != nil {
t.Fatalf("buildlibrary.Open: %v", err)
}
t.Cleanup(func() { _ = lib.Close() })
if err := lib.Truncate(context.Background()); err != nil {
t.Fatalf("truncate: %v", err)
}
return lib
}

Expand Down
16 changes: 9 additions & 7 deletions deploy/helm/ro-builder/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,9 @@ happens outside the chart's lifecycle.

See `values.yaml` for the full schema and defaults. Key sections:

- `api.*`: image, resources, persistence, probes. Always `replicas: 1`
with `strategy: Recreate` (SQLite is single-writer).
- `api.*`: image, resources, persistence, probes. `api.replicas` is
configurable; `api.autoscaling` enables HPA. The default strategy is
`RollingUpdate` (the API is stateless against Postgres).
- `sidecar.*`: image, replicas, workers, autoscaling, probes. Stateless.
- `sidecar.calcBackend`: calc backend selection; see
`calc-sidecar/src/backends/registry.ts`. Use `stub` for chart smoke
Expand All @@ -56,9 +57,10 @@ See `values.yaml` for the full schema and defaults. Key sections:
- `ingress.*`, `networkPolicy.*`, `metrics.serviceMonitor.*`: gated,
default off.

## Why replicas: 1 on the API?
## Database

The buildlibrary is SQLite-backed. SQLite is single-writer and lives in
a process-local file. Multiple API pods would either fight for the PVC
lock (RWO mode) or, if they shared one, corrupt the DB. Horizontal scale
unblocks once the buildlibrary migrates to Postgres.
The API is stateless and connects to Postgres via `DATABASE_URL`, read
from the `ro-builder-env` Secret. `postgresql.enabled` (default `true`)
deploys a bundled in-cluster Postgres StatefulSet. To use a managed
Postgres instead, set `postgresql.enabled: false` and point `DATABASE_URL`
at the external host in your sealed secret.
4 changes: 4 additions & 0 deletions deploy/helm/ro-builder/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -54,3 +54,7 @@ across Deployment, Service, ServiceAccount, etc.
{{- define "ro-builder.sidecarName" -}}
{{- printf "%s-sidecar" (include "ro-builder.fullname" .) -}}
{{- end -}}

{{- define "ro-builder.postgresName" -}}
{{- printf "%s-postgres" (include "ro-builder.fullname" .) -}}
{{- end -}}
Loading
Loading