Running vibey on Kubernetes¶

[!IMPORTANT] Work in progress — not a final product: Running vibey on Kubernetes is actively being developed and is not a final product. Folks will be notified on the Discord server once the product is finally done.

This guide takes vibey from a laptop process to a long-lived deployment: a containerized worker, a Helm chart, an in-cluster PostgreSQL, and queue-depth autoscaling via KEDA. Everything below was verified on minikube (Kubernetes v1.35.1). The chart is written so the same install works against a managed Postgres on AKS/EKS/GKE by setting postgres.enabled=false and dsn.existingSecret (there are no per-cloud values presets yet), but only the local path has been exercised end to end so far. The design decisions behind the image, chart, operator, and autoscaler are recorded in ADR-0025 and ADR-0026.

What works today, and what does not¶

Be clear about this before you install anything:

• The worker runs, applies migrations, claims jobs, and autoscales.
• Every engine ships in the image; none is configured by default. Since ADR-0037 the one vibey wheel carries all three runners, so the image puts claudeloop, codexloop, gptossloop and qwenloop on PATH beside vibey (CI's image job asserts every console script resolves). The chart still defaults to worker.provider: scripted with no worker.engines and no keys, and that is the install CI deploys. Its worker logs no recorded conformance for ..., naming the engines that have none: correct, not a misconfiguration, because an engine without a key can never pass conformance and is never selected. Engine-driven (BUILD) jobs therefore do not run on a default install.
• A binary on PATH is not yet a working engine. Shipping the runners is not the same as running them headless. codexloop drives an external codex binary, which the image does not carry. claudeloop runs through the Claude CLI bundled inside claude-agent-sdk, but claudeloop doctor looks for claude on PATH only, so it reports the CLI missing in the image. That per-runner verification is runbook 16's Phase 0 and runbook 05 item 1.
• Engine authentication in a pod is by API key only. Subscription login is an interactive TTY flow and does not exist in a cluster. The chart reads the keys from a Secret you create: set engineAuth.existingSecret to its name and list engineAuth.keys as {name, key} pairs (environment variable name, key inside the Secret), for example {name: ANTHROPIC_API_KEY, key: anthropic}. They are injected into the worker Deployment only. The variables each engine looks for are ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN (claudeloop); and OPENAI_API_KEY, AZURE_OPENAI_API_KEY, or CODEX_API_KEY (codexloop). Declare the engines you mean to use in worker.engines: vibey doctor --cluster --engines <the same list> then reports engine-auth as FAIL for any of them without a key (see Preflight from inside a pod).
• The operator is implemented, but off by default. vibey operator (pip install 'vibey-engine[operator]') runs kopf handlers that create projects and apply spec.answers through the same application services vibey new / vibey answer use, then reconcile VibeyProject status every 15s. The chart does not install it unless you set operator.enabled=true (step 7 below); without that flag, creating projects and answering gates is still vibey new / vibey answer, run inside a pod or against the database.

Prerequisites¶

• Docker (or colima) and minikube, helm, kubectl.
• For autoscaling, KEDA installed in the cluster (step 5).

1. Build the image into the cluster's daemon¶

The chart defaults to image.pullPolicy: Never and tag vibey:dev, because a locally built image has never been pushed anywhere and Always would send kubelet hunting a registry that has never seen it. Build directly into minikube's Docker daemon:

minikube start -p vibey
eval $(minikube -p vibey docker-env)
docker build -f deploy/docker/Dockerfile -t vibey:dev .

No prebuilt image is published — not to ghcr.io, not to Docker Hub. CI's image job builds amd64 and arm64 on every push and pull request and contract-tests the amd64 build, but it never pushes either. The only ghcr.io artifact the project publishes is the Python distribution (ghcr.io/the-vibey-project/vibey/python), which is a wheel and an sdist pushed with oras, not a runnable image. For any cluster other than minikube, build and push to a registry you control, then point the chart at it:

docker buildx build -f deploy/docker/Dockerfile \
  --platform linux/amd64,linux/arm64 \
  -t <registry>/vibey:<tag> --push .
helm install vibey deploy/helm/vibey -n vibey --create-namespace \
  --set image.repository=<registry>/vibey --set image.tag=<tag> \
  --set image.pullPolicy=IfNotPresent

The image is two-stage on purpose: the runtime layer carries no compiler, no uv, and no pip (the base image's pip is deleted), so a compromised engine session inside a worker finds nothing it can install with. apt-get remains, because the runtime stage uses it to install git and tini, but it needs root and the pod runs as uid 10001 with allowPrivilegeEscalation: false. The entrypoint is tini -g -- vibey (see step 6). Migrations ship inside the image, so an install never depends on someone running SQL by hand first. CI asserts each of these against the amd64 build: the entrypoint runs, id -u is 10001, none of uv pip pip3 gcc cc is on PATH, and /app/migrations/*.sql is non-empty.

2. Install the chart¶

helm install vibey deploy/helm/vibey -n vibey --create-namespace

This creates a ServiceAccount, a worker Deployment, a worktree PVC (mounted at /work, the worker's working directory), a vibey-vibey-dsn Secret, and a single-replica PostgreSQL StatefulSet behind a headless Service. An init container waits for Postgres so the failure mode is "pod pending" rather than "CrashLoopBackOff with a stack trace", and the worker applies migrations itself at startup.

The built-in Postgres is development only — postgres.password defaults to vibey in plain values. For anything real, set postgres.enabled: false and point dsn.existingSecret at a Secret with two keys (ADR-0055):

• dsn (or the key named by dsn.existingSecretKey) is the application role's DSN. The chart injects it as VIBEY_PG_URL into the worker and, when enabled, the operator.
• The owner's DSN, under the key dsn.existingSecretMigrateKey names. It is mounted only into the migrate init container, which runs vibey migrate: it applies migrations, creates the application role if it is missing, grants it exactly the declared privileges, and fails the pod if that role could still rewrite the ledger.

dsn.existingSecretMigrateKey is empty by default, which makes a single-DSN install, so an existing Secret keeps working after an upgrade until you name an owner key. The worker then migrates as the one role, and vibey doctor --cluster fails ledger-guard until the roles are split (database roles). Use a fully qualified host or an IP address: KEDA reads the application's DSN from another namespace.

The built-in Postgres (postgres.enabled: true) does this for you: postgres.user is the owner and postgres.appRole (default vibey_app) the application role. Each surface database (postgres.additionalDatabases) is owned by a login role of the same name whose password is postgres.additionalDatabasePasswords.<name>; the postgres container creates it and hands it its database on every start, so an existing install is converted on upgrade and no surface pod holds the owner's credentials.

Every connection to the built-in Postgres authenticates with scram-sha-256, local and remote alike (sub-doctrine 10.j, ADR-0061). The chart ships the server's pg_hba.conf as the <release>-postgres-hba ConfigMap and points hba_file at it on every start, so an install created before this rule follows it after an upgrade. It also sets password_encryption=scram-sha-256. The method is not a value you can change. A managed instance must meet the same rule: require scram-sha-256 for every role vibey and the surfaces connect as, with no trust, md5 or clear-text password rule that could match them. See SECURITY.md §7 for the lines; vibey doctor --cluster's local-auth does not pass until they are set.

The wait-for-postgres init container is rendered only for the built-in Postgres. Against a managed instance the worker connects directly at startup, so an unreachable DSN shows up as CrashLoopBackOff rather than a pending pod. Check vibey doctor --cluster (database and dsn-host) first.

3. Create a project¶

A worker with nothing to do would normally exit, which in a Deployment is a restart loop that ends only when a human creates a project — and the crash counter makes a healthy worker look broken. The chart therefore sets --wait-for-project 15, so the worker parks and polls:

no project yet; polling every 15s

Create one from inside the cluster:

kubectl exec -n vibey deploy/vibey-vibey-worker -- \
  vibey new demo --repo /work/demo --max-cycles 1

Set worker.project explicitly. Left empty, the worker binds to whichever project was created most recently — convenient on a laptop, a footgun in a cluster the moment a second project exists. The value is the project UUID, not its name. vibey new prints it on creation; there is no list-projects command yet, so otherwise read it from the database (SELECT id, name FROM project;). The chart does not enforce this — with the value empty it omits --project and installs anyway — so it is on you:

bash helm upgrade vibey deploy/helm/vibey -n vibey \ --set worker.project=<uuid>

4. Watch it work¶

kubectl logs -n vibey deploy/vibey-vibey-worker -f

sigterm handler registered
worker started: project=demo engines=all parallelism=2 provider=scripted
processed one job

kubectl logs interleaves stdout and stderr. A current worker also writes per-iteration diagnostics to stderr — drive[N] iter=M calling run_once, drive[N] iter=M run_once returned worked=…, and drive[N] iter=M reap done, waiting for notify — so the real log is noisier than the sample above. Those lines are expected.

Preflight from inside a pod¶

A worker can start, log worker started, report Ready, and still do no work: its worktree volume is not writable, an engine has no credentials, or its DSN is one the autoscaler cannot resolve. vibey doctor --cluster checks that wiring from inside the pod:

kubectl exec -n vibey deploy/vibey-vibey-worker -- vibey doctor --cluster

doctor runs as a separate process, so it cannot see the worker's own command line. Pass it the worker's --engines and --provider (the chart's worker.engines and worker.provider) so the engine-auth check judges the engines the worker will actually use:

kubectl exec -n vibey deploy/vibey-vibey-worker -- \
  vibey doctor --cluster --engines claudeloop,codexloop --provider claudeloop

Every runner is on PATH in the image, so a binary being present says nothing about intent, and the check does not treat it as intent.

It prints one PASS or FAIL line per check and exits non-zero if any check fails:

Check	Passes when
dsn-host	the DSN host is fully qualified, an IP address, or localhost, so KEDA's operator in another namespace can resolve it
non-root	the process uid is not 0
workspace-writable	the working directory (/work in the chart) accepts a write
engine-auth	every engine named by --engines, plus claudeloop under --provider claudeloop, is on PATH and has one of its API-key variables set (gptossloop and qwenloop take none). With neither flag nothing is required: it passes and reports which engines in the worker's default pool have a key, so a default install says plainly that no engine-driven job can run
database	the DSN connects
migrations	every file in /app/migrations is recorded in schema_migration; runs only when database connected

Run it whenever a worker reports Ready but does no work.

5. Autoscaling with KEDA¶

KEDA is a cluster-wide operator and is not assumed, so the ScaledObject is off by default and inert without it.

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda -n keda --create-namespace --wait
helm upgrade vibey deploy/helm/vibey -n vibey --set keda.enabled=true

The trigger is the claimable-work query — ready, due now, not of an abandoned project, and not blocked behind an unsatisfied dependency — deliberately mirroring JobRepository.claim's SELECT arm (tests/infrastructure/db/test_keda_scaler_query.py pins each exclusion to the claim's own). Scaling on raw queue depth would start workers for jobs nothing can claim yet.

Measured behavior on minikube with maxReplicas: 4:

Event	Observed
20 claimable jobs enqueued	0 → 4 replicas in ~8s
queue drained	4 → 0 after the 300s cooldownPeriod
pod receives SIGTERM	exits in 4-5s

Note that the N→0 transition happens in one step. KEDA's deactivation path sets the replica count directly and bypasses the HPA scaleDown behavior policy entirely. That policy only smooths HPA-driven scaling between minReplicas and max — do not read it as protection against losing several in-flight sessions at once.

6. Scale-in and long sessions¶

Engine sessions run for minutes to hours, so terminationGracePeriodSeconds defaults to 7200. That is a ceiling for one long in-flight turn, not an expected shutdown time.

On SIGTERM the worker drains: it finishes the job in hand and claims no more, then exits.

draining on SIGTERM: finishing in-flight job, claiming no more

An idle worker therefore exits in seconds. Only a worker genuinely mid-turn uses any meaningful part of the grace period.

Two details make this hold in the window before the worker is fully up (ADR-0026):

• tini is PID 1. The image's entrypoint is /usr/bin/tini -g -- vibey. Linux discards a signal sent to PID 1 while its disposition is still the default, and a Python interpreter needs hundreds of milliseconds to start and install a handler. On minikube, a pod deleted a fraction of a second after its container started was observed sitting out the entire 7200s grace period, still claiming jobs. tini is ready in microseconds and forwards the signal; -g sends it to the whole process group, so an engine subprocess the worker started is signalled too.
• A SIGTERM latch catches the startup window. vibey arms a small handler before its other imports whose only job is to remember that SIGTERM arrived. Once the event loop is running and the real drain handler is installed, the worker checks the latch; if it fired, the worker logs SIGTERM arrived during startup; draining immediately and claims nothing.

CI's drain contract deletes a worker pod and requires it to terminate in under 60s. It records the container's start time, because a delete that lands during boot tests the startup race and a delete after boot tests the steady-state drain; both must pass.

7. The kopf operator (optional)¶

The chart can also install a cluster-scoped operator that reconciles VibeyProject custom resources, so a project is a CR instead of a kubectl exec command:

helm upgrade vibey deploy/helm/vibey -n vibey --set operator.enabled=true

This installs, in addition to the worker:

• the VibeyProject CRD (vibeyprojects.vibey.dev, short name vp), annotated helm.sh/resource-policy: keep so helm uninstall never deletes a project mid-BUILD along with it. Set operator.installCRD=false if another release already owns the CRD.
• a ClusterRole scoped to vibeyprojects/vibeyprojects/status (list, watch, get, patch, update — deliberately no delete) plus create on events, list, watch on customresourcedefinitions and namespaces (kopf's discovery), and list, watch, patch, get on kopfpeerings/clusterkopfpeerings for kopf peering.
• a single-replica operator Deployment (strategy.type: Recreate; two operators patching the same CR is a race with no upside at this scale). Set operator.watchNamespace to scope it to one namespace instead of the cluster.

Create a project by applying a CR instead of vibey new:

apiVersion: vibey.dev/v1alpha1
kind: VibeyProject
metadata:
  name: demo
spec:
  repo: /work/demo
  maxCycles: 10
  maxCycleDollars: 25
  engines: [claudeloop]
  answers:
    6f1c2a4e-0000-4000-8000-000000000000: { choice: "yes" }

kubectl apply -n vibey -f vibeyproject.yaml
kubectl get vibeyprojects -n vibey

Keys under spec.answers are gate UUIDs — read them from status.openGates[].gateId — and each value is the same object vibey answer sends: {choice: …} for deployment and triage gates, {verdict: …} for review gates, and {<question_id>: <answer>, …} for interview gates.

The CR also accepts maxCycleTurns and skillsContext: {mode, budget, timeout_seconds} (mode is off, shadow, or inject; budget is 1,000–32,000, default 6,000). gates ({timeout_seconds, kill_grace_seconds, isolate_python_env, env_allow}) and engineEnvironment ({allow, engines: {<engine id>: [...]}}) declare what a gate command and an engine session may see of the worker's environment, the same objects as vibey.toml's [gates] and [engine_environment]; a forbidden entry (VIBEY_*, PG*, a DSN) is refused before the project is created. spec.engines is restricted by the CRD schema to the known engine ids (claudeloop, codexloop, gptossloop, qwenloop, claudeloop-local). The worker accepts --provider gptossloop (chart value worker.provider; qwenloop is still read as gptossloop, ADR-0064) for the sovereign DESIGN provider. That provider talks to a local Ollama over HTTP rather than running the gptossloop binary (which the image does ship), so it needs a model server the pod can reach: ollama.enabled runs one in the release and points the worker at it (VIBEY_OLLAMA_URL/VIBEY_OLLAMA_MODEL, and GPTOSSLOOP_BASE_URL/GPTOSSLOOP_MODEL for the gptossloop engine). ollama.qwenloopFeature: true also switches qwenloop on, hands it QWENLOOP_BASE_URL/QWENLOOP_MODEL (ollama.qwenModel, default qwen3:14b) and pulls that model too; it is off by default.

The operator creates the project on first reconcile, then re-reconciles every 15s. It applies any new spec.answers through the same gate-answer service vibey answer calls. A key naming an already-answered gate, a key that is not a UUID, or a value that is not an object is recorded in status.ignoredAnswers as {key, reason}, not treated as an error.

Each reconcile writes:

• status.projectId — the guard that stops a re-applied CR from creating a second project;
• status.phase, status.cycle, and status.maxCycles;
• status.openGates — {gateId, kind, prompt} per open gate;
• three conditions: Ready (True/Progressing, or False with AwaitingHuman, Complete, or Abandoned; Unknown/ProjectMissing if the project row is gone), Parked (True with the gate kind CamelCased as the reason, for example BudgetExhausted), and Complete.

kubectl get vp shows Phase, Cycle, Ready, Parked, Reason (the Parked reason), and Age as columns; kubectl describe shows the full status. The operator never deletes a project, so removing the CR does not remove the underlying project or its data.

Troubleshooting¶

helm upgrade fails with lookup <release>-postgres ... no such host. You are on a chart older than the DSN fix. KEDA's operator runs in its own namespace and dials Postgres itself, so the DSN must be fully qualified — a bare Service name resolves only from inside the release namespace, which is why the worker was fine and the autoscaler was not. Set clusterDomain if your cluster does not use cluster.local.

Deployment shows 0/0 but pods are still Terminating and working. Either the worker is ignoring SIGTERM (an image built before the drain landed) or the signal was discarded before the worker could catch it (an image whose PID 1 is Python rather than tini, built before tini became the entrypoint). Both look the same: kubectl reports capacity released while the pods still hold CPU, memory, and Postgres connections and keep claiming jobs. Rebuild from the current deploy/docker/Dockerfile. On a current image the pod's log shows sigterm handler registered at boot and, on delete, either draining on SIGTERM: finishing in-flight job, claiming no more or SIGTERM arrived during startup; draining immediately. If neither appears after a delete, the image is old.

kubectl delete pod --force --grace-period=0 leaves workers running. Force delete removes the pod object without waiting for the container to die, exactly as its warning says. The container keeps running, keeps its database connections, and keeps claiming jobs — invisible to kubectl, since the pod is gone from the API server. Check with:

kubectl exec -n vibey vibey-vibey-postgres-0 -- \
  psql -U vibey -d vibey -c \
  "SELECT client_addr, count(*) FROM pg_stat_activity
   WHERE datname='vibey' AND client_addr IS NOT NULL GROUP BY client_addr;"

Connections from addresses with no corresponding pod are orphans; kill them at the container runtime (docker kill inside minikube docker-env). Prefer a normal delete — the drain makes it fast.

Workers scale up but claim nothing. The scaler counts claimable jobs across all projects, while the worker Deployment binds to one. Work in another project will scale workers that cannot claim it.

A worker is Ready but does no work. Run vibey doctor --cluster inside the pod, with the worker's --engines and --provider (see Preflight from inside a pod).

Values worth knowing¶

Value	Default	Why
image.repository / image.tag	vibey / dev	local build; override for any registry
image.pullPolicy	Never	change to IfNotPresent once the image is in a registry
worker.terminationGracePeriodSeconds	7200	ceiling for one long in-flight turn
worker.waitForProjectSeconds	15	park instead of restart-looping
worker.parallelism	2	concurrent job loops per pod
worker.project	""	set this; empty binds to the newest project
worker.provider	scripted	DESIGN/decompose provider; claudeloop needs its API key, gptossloop a reachable Ollama (qwenloop is read as gptossloop)
worker.engines	""	comma-separated engine allow-list (--engines); empty means all; pass the same list to doctor --cluster --engines
worker.replicas	1	ignored once KEDA owns the Deployment
worker.worktrees.size / storageClass	5Gi / ""	the /work PVC where BUILD worktrees live
engineAuth.existingSecret	""	Secret holding engine API keys
engineAuth.keys	[]	{name, key} pairs mapped to env vars on the worker
keda.minReplicas / maxReplicas	0 / 4	scale to zero when idle
keda.pollingInterval	15	seconds between scaler queries
keda.cooldownPeriod	300	delay before deactivating to zero
clusterDomain	cluster.local	only change on a custom --service-dns-domain
postgres.enabled	true	dev only; use dsn.existingSecret for managed
postgres.storage	8Gi	dev Postgres PVC
dsn.existingSecret	""	Secret holding a managed instance's DSN
dsn.existingSecretKey	dsn	key inside dsn.existingSecret
operator.enabled	false	install the kopf VibeyProject operator
operator.watchNamespace	""	empty watches cluster-wide
operator.installCRD	true	disable if another release already owns the CRD