Skip to main content

Dev cutover runbook - flipping traffic to apps/server

Operational runbook for the dev env cutover of the TS backend (apps/server), the terminal step of the Python -> TypeScript strangler-fig migration (M29 -> M42). It turns the M40 cutover plan in design/ts-migration-feasibility.md (§"M40 cutover plan", plus the M40-8/M41/M42 addenda) into a concrete, walkable sequence for the dev pair.

This runbook is the plan, not the execution. Nothing here has been run. Every step that touches a live env is gated on Al's explicit in-the-moment go, and no gateway/process restart fires without that go. The migration code is complete and ships dormant on main (server.enabled: false, BLOOM_JOB_WORKER_ENABLED off), so main is safe as-is regardless.

Scope and posture​

  • Env: dev only. No staging tier exists; prod is gated behind the separate paid-staging (M25) decision and is out of scope here. Read the feasibility doc's "staging" steps as dev.
  • Reversible at every step. Both backends read/write the same shared-Postgres tables with no dual-write and no translation layer, so any route served from -server is byte-identical and interchangeable with the same route on -api. A bad flip rolls back with a one-line ingress change; a bad rollout rolls back with server.enabled: false. Decommission is the only one-way door and is not part of this dev runbook.
  • Path-by-path, not all-or-nothing. The two backends coexist over one Postgres, so routes flip one prefix at a time, lowest-risk first, each independently reversible.

The one structural finding: the flip is not a one-line values toggle​

Grounding the chart surfaced the gap the feasibility doc's step 3 implies but the "flip a flag" framing hides:

  • There is no server ingress in the chart. deploy/helm/bloom/templates/ has api-ingress.yaml (backend hardcoded to the -api service) and no server analogue. The ingress values block has api (and, until #728, web) sub-blocks but no server block.
  • So --set server.enabled=true spins the TS server pod up and it will sit Ready and idle - nothing routes traffic to it.

Therefore step 0 of the flip is a reviewed chart PR that adds a server ingress template + ingress.server values block (the api-ingress.yaml analogue), routing chosen path prefixes at the -server service. That is a normal code change through the driver/PR flow, landed and merged before any env toggle. The flip proper starts only after it is on main.

Hard gate before any route flips: the shared-pooler budget​

The constrained resource is server-side Postgres connections through the shared Supabase transaction pooler (Supavisor, port 6543) - its default_pool_size per role+db (Supabase dev default 15; read the live value at cutover from Database -> Connection pooling -> Pool Size). During coexistence both backends run against that one ceiling, so the sum of their pool max values must fit under it.

BackendPool max (server slots)Source
apps/api (Python)10supabase_store.py max_size=10
apps/server (TS)4apps/server/src/db.ts max: 4
Coexistence sum14worst case: all 14 client conns mid-transaction
Pooler ceiling~15 (dev tier default)reserve ~3 for superuser/setup/migration

14 under 15 is one slot of headroom - too tight to flip blind. Action, before the first route flips: trim the Python apps/api pool max_size 10 -> 8 (sum -> 12, ~3-slot margin). The TS side stays max: 4 (already the deliberately small pool, and it carries the new backend). Confirm the dev project's actual Pool Size at cutover: if materially above 15 the trim can relax proportionally; if at or below 15 the trim is mandatory. Sockets (HTTP/SSE connections) are not pooler-constrained - an SSE stream holds a socket but no Postgres slot (M32 finding).

Composition-root wiring is a cutover step, not a port​

M41/M42 ported and parity-proved every orchestrator entry point and every leaf-handler seam body, but app.ts still injects the not-ported defaults because the root does not yet assemble the live provider factory / engine provider / reviewer / design surface / production messenger. It does not need to while dormant (those defaults are never reached). Assembling the live subsystems into app.ts (alongside the already-landed M40-6 messenger stack) is the orchestrator's slice of the "enable the server workload" step - no new port to write first, it is wiring. This lands as part of enabling the workload, under review, before the worker is armed.

Flip sequence (dev)​

Run top-to-bottom. Each step is reversible; do not proceed to the next until the current one is verified. Every env-touching step waits for Al's explicit go.

  1. Land the server-ingress chart PR (above) and the composition-root wiring PR. Merge to main; CI green. No env touched yet.
  2. Trim the Python pool (supabase_store.py max_size: 10 -> 8) and confirm the dev pooler Pool Size. This is the hard gate - do it first.
  3. Enable the server workload in dev. server.enabled: true in the dev overlay, image.tag pinned to the CI-built SHA. The chart mounts the same bloom-api-secrets Secret and reads the same BLOOM_* config the Python API reads (no new secret plumbing). The server comes up serving /healthz (liveness) and /readyz (readiness proves DB connectivity), still with no gateway route pointed at it. Verify both probes green before continuing.
  4. Flip dev path-by-path. Add the server ingress routes most-specific-prefix first, flipping the lowest-risk surface first: read endpoints -> account-scoped -> admin surfaces -> webhook ingress -> orchestrator-coupled routes. Leave any still-deferred path on -api. Each prefix flip is one ingress backend change, independently reversible.
  5. Arm the worker. Flip BLOOM_JOB_WORKER_ENABLED on so the JobRunner drains against the now-complete orchestrator (default coordination engine). Before this, the worker is constructed but unstarted (started stays false).
  6. Prove parity end to end (below) and watch the live pooler connection count against the step-1 budget under coexistence load.

Parity proof​

Two layers of proof, both already the gates that passed every migration slice.

Offline / cross-runtime harness (run from apps/server)​

# Read + write parity: identical request/mutation matrices against BOTH runtimes over one
# shared Postgres, diffing status + content-type + raw body bytes + persisted rows.
# Needs docker (postgres:16-alpine) or an empty DB via BLOOM_PARITY_DB_URL, plus uv + pnpm.
pnpm run parity
BLOOM_PARITY_DB_URL=postgres://... pnpm run parity     # against an existing empty DB
BLOOM_PARITY_VERBOSE=1 pnpm run parity                 # child-process (uvicorn + tsx) logs

# LLM record/replay parity - fully offline, needs only uv (no docker/db/network).
pnpm run parity:llm

# SSE/streaming parity - offline virtual-clock phases + a live end-to-end phase
# (live phase needs docker or BLOOM_PARITY_DB_URL, like `pnpm run parity`).
pnpm run parity:sse
BLOOM_PARITY_SSE_OFFLINE=1 pnpm run parity:sse         # offline SSE phases only

# Async orchestrator LIFECYCLE ordering parity is NOT a separate live harness - it is
# offline goldens generated from the real Python, replayed through the TS twins under vitest:
pnpm run test                                          # includes parity/orchestrator_async/*

Exit 0 on the harness = every case byte-identical; any diff prints in full and exits 1. The parity/orchestrator_async/ suite (M41/M42) diffs each entry point's and leaf handler's side-effect ordering trace against the Python oracle entry-for-entry.

Live lifecycle exercise (checklist step 4)​

Beyond the canned harness, drive a real project lifecycle through the flipped dev backend and diff the observable outcomes against a Python-served baseline:

webhook -> enqueue -> orchestrator worker -> VCS/messenger side effects -> SSE

This now exercises the real leaf handlers (M42), not stubs. Watch the pooler count under this load against the step-1 budget.

Rollback​

  • Bad route flip: point that path's ingress backend back at -api (one-line change). Byte parity + shared Postgres means no data to unwind.
  • Bad server rollout: server.enabled: false - the workload drops out, all traffic on -api.
  • Worker misbehaving: BLOOM_JOB_WORKER_ENABLED off - the drain loop stops; producer stays live, consumer dormant.
  • The graceful-drain-on-stop divergence (below) is safe under rollback: JobRunner.stop() awaits in-flight dispatches to completion rather than cancelling them; at-least-once delivery + the event_key idempotency fence make any re-delivery safe.

Carried forward / out of scope for the dev flip​

  • The langgraph coordination engine is deferred, not blocking. The shipped reconcilePlan is the byte-for-byte default Coordinator.reconcile; the non-default BLOOM_COORDINATION_ENGINE=langgraph path (langgraph_coordinator.py) stays Python-only and un-migrated behind its non-default flag. It differs from the default (its _classify omits the awaiting-client skip guard), so it was never in scope. The dev flip runs the default engine.
  • The one carried behavioral divergence is the M40-7 graceful-drain-on-stop above - strictly safer under the at-least-once + idempotency-fence contract, invisible to the differential harness. No new divergence since.
  • Prod is out of scope. It repeats steps 1-5 against the prod project (with its own re-derived pooler budget), and only after the paid-staging (M25) decision lands. Not part of this runbook.
  • Decommissioning apps/api (remove api ingress, scale to zero, delete the api Deployment/Service/ConfigMap, retire the CI build, restore the surviving pool max) is the one irreversible door - staged last, behind a clean soak, with the Python oracle kept as the rollback anchor. Not part of this dev runbook; since executed (#806-#808), with the oracle preserved in git history at d69f789.