Runbook: server mode — Kubernetes everywhere (minikube → AKS/EKS/GKE)

Status (2026-09-18): landed on minikube (PRs #73, #74, #76, 2026-08-21; ADR-0025, ADR-0026) — image, Helm chart, KEDA ScaledObject, kopf operator + VibeyProject CRD, vibey doctor --cluster, docs/guides/kubernetes.md. Since ADR-0037 (PR #234) the image carries every runner, so "engines in the image" is done as packaging; what stays open in item 1 is making each one work headless there (runbook 16, Phase 0). Also open: the server Deployment (depends on 12), item 6 (cloud presets), PodDisruptionBudget, and the paid live and AKS/EKS/GKE verification runs.

Goal

vibey runs as a long-lived server deployment, not a MacBook process: containerized workers on Kubernetes, packaged as a Helm chart, autoscaled by queue depth via KEDA, operated by a kopf-based operator, runnable locally on minikube and hosted on AKS, EKS, and GKE. Engine sessions run against LLM provider APIs/SDKs (API-key auth), since interactive subscription login doesn't exist in a cluster.

Current state (verified)

  • vibey worker is a solid headless process already: Postgres queue (FOR UPDATE SKIP LOCKED), leases + reaping, LISTEN/NOTIFY, per-kind leases, advisory-locked integrates — worker death is already survivable (idempotent replay). This is 90% of being cluster-ready.
  • vibey ships as a two-stage, non-root image (deploy/docker/Dockerfile, tini as PID 1 so SIGTERM reaches the worker's drain latch — ADR-0026) and a Helm chart (deploy/helm/vibey/: worker Deployment, in-cluster Postgres, KEDA ScaledObject, operator Deployment, VibeyProject CRD). CI builds the image for amd64 + arm64 with five image contracts and runs a minikube job with four cluster contracts (projectless worker parks, in-cluster project is picked up, the ScaledObject reconciles against real Postgres, a worker drains promptly on SIGTERM).
  • The image carries every runner. ADR-0037 made the one vibey wheel ship all three, and the Dockerfile copies each package root, so claudeloop, codexloop, gptossloop and qwenloop are on PATH in every pod (CI's image job asserts every console script). The chart still defaults to --provider scripted with no engine keys, and no runner has yet completed a session in-cluster: on a laptop they still authenticate via subscription login.
  • infrastructure/container/runtime.py holds container runtime helpers; infrastructure/cluster_preflight.py backs vibey doctor --cluster and maps each of the two hosted-model engines (not gptossloop or qwenloop) to the API-key environment variables it accepts. Its engine-auth check judges the engines the worker is told to use (doctor --cluster --engines … --provider …, the worker's own flags), not every binary on PATH.

Design

  1. Images: one vibey image (uv-built, non-root, distroless-ish). The second vibey-engines image first designed here is moot since ADR-0037: the one wheel carries the loop runners, so the one image does too. Engine API-key mode: each runner must support ANTHROPIC_API_KEY/OPENAI_API_KEY auth (runner-side work item where missing — subscription login is a TTY flow).
  2. Helm chart (deploy/helm/vibey/): Deployments for worker (N replicas) and server (the FastAPI ingress from workstream 12 — webhooks, API, health); CloudNativePG-or-managed-Postgres option; Secrets for DSN + engine keys; ConfigMap for project defaults; PodDisruptionBudget; liveness = DB ping, readiness = migration status.
  3. KEDA: ScaledObject on the worker Deployment using the postgresql scaler — query = ready-job count due now (the claim query's SELECT arm); min 0 / max N replicas. Long engine sessions are protected from scale-in by the worker's own SIGTERM drain -- finish the job in hand, claim no more -- bounded by terminationGracePeriodSeconds. Landed as a signal handler in the worker rather than a preStop hook: a preStop script cannot tell a running worker to stop claiming, and the drain has to be a property of the process, not of the pod spec.
  4. kopf operator (landed at src/vibey/infrastructure/operator/, run as vibey operator; decision logic in application/operator_projection.py): a VibeyProject CRD — spec holds repo URL, budget caps, engine allow-list; the operator runs vibey new, watches phase, surfaces parks as CR status conditions + Kubernetes Events, and applies answers written into the CR (spec.answers) through the same service vibey answer uses. A 15 s level-triggered timer reconciles status. AKS/EKS/GKE-specific bits (workload identity per cloud) are to live in values presets: values-aks.yaml, values-eks.yaml, values-gke.yaml. These presets do not exist yet, and both values.yaml's header comment and the Kubernetes guide say so.
  5. Worktrees in-cluster: a PVC per worker for git worktrees; repos cloned via deploy keys mounted as Secrets.
  6. Keep-awake is a non-problem here (10 covers desktops).

Work items

  1. Engine API-key auth across the three runners (per-runner work items). Open: the runners ship in the image (ADR-0037), but none is proven headless there yet — see 16, Phase 0.
  2. Dockerfiles + CI image builds (multi-arch: arm64 + amd64). Done (#73); since ADR-0037 the one image carries the runners too, so there is no separate engines image to build.
  3. Helm chart + kind/minikube smoke test in CI (helm install → seed a scripted-engine project → DONE local). Chart and minikube job done (#73); the CI contracts stop at pickup, not DONE. No server Deployment or PodDisruptionBudget yet.
  4. KEDA ScaledObject + scale test (enqueue 20 jobs → replicas rise → drain → scale to zero). Done (#73); CI asserts reconcile and SIGTERM drain.
  5. kopf operator + CRD + park-to-condition flow + answer application. The same operator later carries the plan-drift reconcile loop -- see 17-plan-drift-reconciliation.md, which builds directly on this CRD and its condition/Event plumbing. Done (#76).
  6. Cloud presets: AKS/EKS/GKE values + workload-identity wiring (reuses workstream 03 tenants). Open.
  7. vibey doctor --cluster: in-cluster preflight (DB, secrets, engines). Done (#74).
  8. Runbook doc: docs/guides/kubernetes.md. Done.

Verification

  • minikube: helm install → full greeter run with scripted engines → DONE, zero manual steps; KEDA scales 0→N→0 observed. Partly done: the CI minikube job proves install, pickup, reconcile and drain; a full run to DONE is not in CI.
  • One paid live greeter on minikube with API-key claudeloop. Open.
  • AKS + EKS + GKE (open): chart installs, a scripted-engine project completes on each (managed Postgres), teardown clean.

Needs from operator

The minikube path needs nothing new (CI runs it). Remaining: the 03 cloud tenants and LLM API keys for API-key engine mode.

Risks

  • Engine subscription-vs-API pricing differs materially — budget caps are mandatory in cluster values (the brake now reads real spend).
  • Scale-in during a 2h implement session. Tested with a forced drain, and it failed the first time: the worker had no signal handling at all, so a scaled-in pod kept processing jobs 77s after SIGTERM, and five "terminated" pods still held live Postgres connections while the Deployment reported 0/0 — scale-to-zero freeing nothing. Fixed by the SIGTERM drain above; pods now exit in 4-5s. Re-test this on every change to the worker loop: it is the failure mode that looks green from kubectl while being completely broken.
  • CRD answer channel is a second write path to gates — it must call the same application service as vibey answer (single choke point).