Skip to main content

ADR 0001: The engine-runner stays on a dedicated VM (not a k8s workload)

  • Status: accepted (M24-6)
  • Date: 2026-08-10
  • Deciders: M24 milestone (Cloud Infra: Kubernetes-Native Dev + Staging Readiness)

Context​

M24 moves the Bloom app onto Kubernetes: the api/web/swarm workloads deploy to a single-node k3s cluster on the dev VM (bloom-server.exe.xyz) via the bloom Helm chart, with CI-driven deploys and sealed-secrets (M24-1..5).

The engine-runner (docs/engineering-engine.md) is the one service left outside that model: a systemd unit on a dedicated worker VM (bloom-engine.exe.xyz), reached by the app over the owner's tailnet, executing Claude Code coding jobs. This ADR answers M24-6's question: does the runner become a Kubernetes workload, or does it stay a VM exception?

Two properties of the runner drive the answer:

  1. It executes untrusted, generated code. A run installs dependencies and runs tests written by a model. M17 placed that on a separate VM specifically so it never shares a kernel with Bloom's GitHub App key, database DSN, and session secrets ("blast radius").
  2. It is deliberately single-flight. One run at a time; a busy runner answers 429 + Retry-After and Bloom's classifier treats that as a retryable pause. There is nothing to schedule, scale, or bin-pack.

Decision​

The engine-runner stays a systemd service on its dedicated VM. The existing deploy path (deploy/engine-runner/deploy.sh + bloom-engine-runner.service) is unchanged.

What M24-6 adds is the tailnet bridge from the k8s side: the Helm chart's api.hostAliases value pins the runner's MagicDNS name to its tailscale IP in the api pod — the exact analog of the pins docker-compose.yml has carried in extra_hosts since M17-7 — so BLOOM_ENGINE_RUNNER_URL keeps working verbatim after the M24-5 compose→k3s cutover.

Options considered​

A. Runner as a workload on the existing dev k3s cluster — rejected​

The dev cluster is a single node on the same VM as the api, the sealed-secrets controller private key, and every app Secret. Scheduling the runner there would move arbitrary-code execution from a separate VM onto the shared kernel that M17 deliberately kept it away from — a container boundary (namespaces + cgroups) is strictly weaker than the VM boundary we have today, and generated code running npm install/pytest is exactly the workload container escapes are built from. This option reduces isolation to gain a uniform deploy tool. Rejected on security grounds regardless of convenience.

B. Runner as a workload on a dedicated k8s node (the worker VM joins the cluster) — rejected​

Isolation-equivalent on paper (taint the node, pin the pod), but every k8s benefit degenerates at n=1 dedicated node:

  • Scheduling/scaling buy nothing. The pod is node-pinned and single-flight; a Deployment with replicas: 1 on one tainted node is systemd with more moving parts.
  • Tailnet connectivity gets harder, not easier. Today the runner binds the VM's tailscale address directly (BLOOM_RUNNER_HOST from tailscale ip -4), unreachable from the public internet by construction. In-cluster, the pod would need the Tailscale Kubernetes operator/sidecar — which needs NET_ADMIN + /dev/net/tun (or userspace-proxy mode) — directly against the chart's hardening posture (runAsNonRoot, capabilities: drop: [ALL], allowPrivilegeEscalation: false). We evaluated the official tailscale-operator rather than hand-rolling; the tool is fine — the requirement (a privileged network sidecar next to untrusted code) is what's wrong.
  • Secret exposure widens. The Claude subscription OAuth token lives only in a 0600 EnvironmentFile on the worker VM. As a cluster Secret it becomes readable to anyone with namespace read and lands in the control plane's datastore on the shared dev VM.
  • Workspace persistence is a wash. /home/exedev/bloom-workspaces on the VM vs a local-path PV on the same disk — no durability gain, one more abstraction.
  • Cost. k3s control-plane traffic, flannel-over-tailnet, and node upkeep on the worker VM, against M24's $0 constraint, for zero functional gain.

C. Keep the dedicated VM, bridge the tailnet from the chart — accepted​

Preserves the M17 isolation model and the working deploy path; the only genuinely new problem M24 created — the app moved into pods, can it still reach the runner? — is solved by two boring, standard mechanisms (below). Revisit triggers are explicit (see Consequences).

Tailnet networking: how api pods reach the runner​

Two hops, both preserved without privileged pods or a Tailscale sidecar:

  1. Routing. The dev VM (the k3s node) runs tailscaled, so the node routes 100.64.0.0/10 via tailscale0. Pod egress to any address outside the cluster/service CIDRs (10.52.0.0/16 / 10.53.0.0/16 — see the CIDR pin; no overlap with tailscale's CGNAT range) is masqueraded by flannel to the node's own address on the egress interface — here the node's tailnet IP. The runner sees the dev VM as an ordinary tailnet peer, exactly as it does compose traffic today.
  2. Names. Cluster DNS (CoreDNS) does not consult the tailscale resolver, so *.ts.net MagicDNS names are pinned per environment via api.hostAliases (values-dev.yaml) — the same names and IPs as docker-compose.yml's extra_hosts (tailscale IPs are stable for the life of a node).

The runner's own exposure is unchanged: bound to its tailscale address on port 8700, bearer-authed /health and /metrics, never internet-facing.

Verify from the cluster (a 401 proves tailnet reachability - the runner is up and demanding its bearer token; 200 needs the token):

# routing (hop 1): in-cluster pod -> runner tailscale IP
kubectl -n bloom run tailnet-check --restart=Never --image=curlimages/curl --command -- \
  curl -sS -o /dev/null -w '%{http_code}\n' --max-time 10 http://100.94.75.123:8700/health

# names (hop 2): the same request by MagicDNS name, pinned the way the chart pins it
kubectl -n bloom run tailnet-name-check --restart=Never --image=curlimages/curl \
  --overrides='{"spec":{"hostAliases":[{"ip":"100.94.75.123","hostnames":["bloom-engine.tail6e7da9.ts.net"]}]}}' \
  --command -- curl -sS -o /dev/null -w '%{http_code}\n' --max-time 10 \
  http://bloom-engine.tail6e7da9.ts.net:8700/health

Both returned 401 from the dev cluster when this ADR was accepted (2026-08-10).

and end-to-end via the app: GET /health on Bloom reports engine.runner.reachable=true when BLOOM_ENGINEER_ENGINE=claude_code is configured.

Consequences​

  • The runner remains the one non-k8s deployable; its deploy path (deploy/engine-runner/deploy.sh, tar-over-ssh + systemd) is untouched — no regression.
  • BLOOM_ENGINE_RUNNER_URL / BLOOM_ENGINE_RUNNER_TOKEN flow to the api pod like every other secret (Actions secrets → sealed-secrets → bloom-api-secrets, M24-4); no special handling.
  • The chart carries a small, documented api.hostAliases seam; staging/prod overlays leave it empty until those environments get an engine-runner story (M25+).
  • Ops for the runner stay VM-shaped (journalctl -u bloom-engine-runner), not kubectl logs. Acceptable: it already exports bearer-authed Prometheus /metrics (#355) into the same observability surface.

Revisit when any of these change the calculus: (a) runs need to scale horizontally (multiple concurrent runners → a real scheduling problem), (b) a hardened runtime class (gVisor/Kata) lands in the clusters, making in-cluster untrusted execution defensible, (c) M25+ standing infrastructure replaces the exe.dev VMs entirely, or (d) the Tailscale operator can attach the tailnet without privileged capabilities beside the workload.