Skip to main content

Deployment

Bloom runs as a Docker container on an exe.dev VM, deployed by GitHub Actions. exe.dev's edge terminates TLS and gives the VM a stable hostname (https://<vmname>.exe.xyz), so there's no reverse proxy, certificates, or DNS to manage. State lives in Supabase (see persistence.md), so the container is stateless and safe to rebuild or restart.

Telegram / GitHub  ->  https://<vmname>.exe.xyz   (exe.dev edge: TLS + stable hostname)
                        -> bloom container  (uvicorn, host 8080 -> container 8000, restart: unless-stopped)
                        -> Supabase          (durable state, already remote)

Continuous deployment (GitHub Actions)​

Two deployment environments, each with its own workflow and GitHub Environment. The API and the web dashboard (Cloudflare Pages) are deployed separately but paired per environment:

EnvAPI workflowWeb workflowTriggerTarget
devdeploy-dev.ymldeploy-web-dev.ymlauto on push to main (+ manual)API on the exe.dev VM; web on the bloom-dashboard Pages project - the always-current, testable live deployment
productiondeploy-production.ymldeploy-web-production.ymlmanual only (workflow_dispatch)API on separate infra (TBD); web on the bloom-dashboard-prod Pages project - both inert until a production API is stood up

Web environments (Cloudflare Pages). Two Pages projects, one per environment: bloom-dashboard (dev) and bloom-dashboard-prod (production). The SPA builds same-origin; the Pages Function reverse-proxies /api to the API named by that project's API_ORIGIN env var (dev defaults to the exe.dev API). Production web is manual-only and inert until a production API exists - then set the prod project's API_ORIGIN, register its OAuth redirect URI, and point the prod API's BLOOM_DASHBOARD_URL / BLOOM_OAUTH_REDIRECT_BASE_URL at it (see deploy-web-production.yml).

Docs site (Cloudflare Pages, M45-5). The documentation site (apps/docs, Docusaurus) is a third Pages surface with the same two-project split: deploy-docs-dev.yml auto-publishes the built site to the bloom-docs project after CI succeeds on main - live at bloom-docs.pages.dev - and deploy-docs-production.yml is manual-only (workflow_dispatch) to bloom-docs-prod. The site is static and API-independent, so there is no API_ORIGIN/OAuth wiring. Unlike the other Pages deploys, the docs deploy fails loud when Cloudflare is unconfigured (missing credentials fail a preflight step; a missing Pages project fails the wrangler publish) - it never silently skips, so a green run always means the linked docs URL actually updated. One-time operator step per environment: create the Pages project before the first run (wrangler pages project create bloom-docs --production-branch=main, and bloom-docs-prod for production). The project names can be overridden with the CLOUDFLARE_DOCS_PAGES_PROJECT / CLOUDFLARE_DOCS_PAGES_PROJECT_PROD Actions variables.

Both workflows do the same two shared steps - render the .env from Actions secrets and variables (scripts/deploy/render-env.sh), then run a target script that knows how to reach that environment's infrastructure. The target script is the only provider-specific piece:

  • dev → scripts/deploy/exedev.sh (SSH to the exe.dev VM; pulls the tagged image)
  • production → scripts/deploy/production.sh - a VM over SSH today, so it delegates to the same tagged-image deploy/rollback path as dev once the production environment has DEPLOY_HOST/DEPLOY_SSH_KEY; with none set it fails fast as "not configured". A different provider is a change to this one file, keeping the same MODE + BLOOM_IMAGE_TAG contract.

GitHub is the source of truth for configuration. The render merges all secrets and variables into .env on each deploy - so you set config in Actions settings, never hand-edit .env on a server. It uses jq's @json so every value (URLs, tokens, anything with $ # " \) is escaped exactly as the app's parser expects. On the k3s dev cluster (M24) the same source of truth holds, but the secrets portion is delivered as Kubernetes Secrets via sealed-secrets instead of a .env - add/rotate runbook in deploy/k8s/README.md#secrets.

Config layering with GitHub Environments: put shared app config at the repository level (e.g. ANTHROPIC_API_KEY, most BLOOM_*); put per-environment overrides on the dev / production environment (they win over repo-level), so production can use a distinct bot token, database, and domain from dev without them colliding. Deploy-only infra uses the DEPLOY_ prefix - it is scoped to its environment and excluded from the rendered .env (along with the injected github_token).

The dev target (exedev.sh) follows exe.dev's SSH pattern but deploys by pulling a pre-built, SHA-tagged image rather than rebuilding on the VM (see Immutable images, deploy-by-tag & rollback below). It streams just the compose file and the rendered .env over the ssh exec channel (exe.dev has no scp/sftp), preserving .env, secrets/, and the rollback bookkeeping; then it logs into the registry, runs docker compose pull + up -d --no-build, and polls http://localhost:8080/health. The VM never builds and needs no source checkout.

The container publishes host port 8080 (it still serves 8000 internally) to avoid exe.dev's default server placeholder on 8000. Point exe.dev's edge at it once: ssh exe.dev share port <vmname> 8080 and ssh exe.dev share set-public <vmname> - then https://<vmname>.exe.xyz forwards to Bloom.

Immutable images, deploy-by-tag & rollback (M15)​

Deploys are build-once, deploy-many: CI builds the API image once and the deploy runs that exact artifact, so a deploy is immutable, reproducible, and reversible.

  • Build & push (CI). The build stage (ci-build.yml) builds the API image and, on a push to main, pushes it to GHCR tagged sha-<12> and latest. PRs build the image to validate it without pushing.
  • Deploy by tag. The deploy checks out the exact commit CI built, computes the same sha-<12> tag, and passes it to the target script, which pulls ghcr.io/hidden-claw/bloom-ai:sha-<12> and runs it with --no-build. docker-compose.yml references image: ${BLOOM_IMAGE:-ghcr.io/hidden-claw/bloom-ai}:${BLOOM_IMAGE_TAG:-latest}; build: stays only as a local/offline fallback.
  • Registry auth. The VM pulls the private image with the workflow's GITHUB_TOKEN (packages: read), passed in and used for a docker login ghcr.io at deploy time - so there is no long-lived registry secret to manage on the VM. (A restart reuses the local image and needs no pull.)
  • Rollback. Each healthy deploy records the running tag in .deployed_tag and the tag it replaced in .deployed_tag_prev on the VM. The Rollback (dev) workflow (deploy-rollback-dev.yml, workflow_dispatch) redeploys .deployed_tag_prev - one click restores the previous image; the pointers swap so the action is itself reversible. Production rolls back the same way via deploy-production.yml's mode: rollback input.
  • Auto-rollback on e2e failure (E2E-7). The post-deploy e2e stage (e2e.yml) tracks a separate .last_green_tag - the last image that passed e2e. When the suite passes it promotes the running image to that pointer; when it fails after an API deploy it runs rollback_green, redeploying .last_green_tag so the dev API is never left broken (the red run + a job-summary note are the notification). This is distinct from the manual rollback above: .last_green_tag only ever advances on an e2e pass, so it stays a known-good target even across consecutive failures. It is idempotent and a no-op until a first green exists. (A web-deploy or manual-dispatch e2e failure is not fixed by an API rollback, so those are skipped.)
  • Retention. After a healthy deploy the VM keeps the current + previous + last-e2e-green image tags (so every rollback target stays local, independent of the registry) and prunes older ones, then reclaims dangling layers - the disk does not grow without bound.

App rollback is not data rollback. Rolling the image back does not revert Supabase schema/data. A deploy that ran a migration cannot be cleanly undone by image alone - keep schema changes backward-compatible (Bloom already creates tables idempotently). A real data-rollback story is out of scope.

One-time setup for CD​

  1. Prepare the VM (once): the deploy directory is created automatically, so this is minimal.

    sudo usermod -aG docker exedev   # so `docker compose` runs without sudo (re-login to apply)

  2. Create an SSH key for CI and authorize it on the VM:

    ssh-keygen -t ed25519 -f bloom-deploy -C "github-actions-deploy" -N ""
    # register bloom-deploy.pub with exe.dev (ssh exe.dev -> ssh-key add) so it can reach the VM

  3. Add the shared app configuration as repository secrets (sensitive: API keys, tokens, DB URL) and variables (non-sensitive: BLOOM_ENV, BLOOM_LLM_MODEL, BLOOM_PUBLIC_BASE_URL, ...) - the keys from .env.example. These are the defaults both environments render into .env.

  4. Set up the dev environment (Settings -> Environments -> dev) with the exe.dev deploy secrets - scoped to the environment and kept out of the rendered .env:

    dev env secretValue
    DEPLOY_HOSTthe VM's SSH host (e.g. <vmname>.exe.xyz)
    DEPLOY_SSH_KEYcontents of the private key registered with exe.dev

    deploy-dev.yml connects as exedev@$DEPLOY_HOST. A push to main then auto-deploys to dev once CI is green.

  5. production environment: create it for later. Put production-specific overrides there (its own bot token, DB, domain). Production is a VM over SSH today, so add DEPLOY_HOST + DEPLOY_SSH_KEY (same shape as dev) and it deploys/rolls back by tag through the shared exedev.sh path; with none set, a manual production run fails fast with a clear "not configured" message. On a paid plan you can also add Required reviewers to either environment for an approval gate; on the free plan the gate is that production is manual-only.

Enabling the production engineer engine (M6)​

BLOOM_ENGINEER_ENGINE selects who implements delegated tickets and defaults to disabled - delegate via labels and let an external swarm or humans do the work, exactly as before M6. Turning on in-process execution is always an explicit opt-in:

  • simulator - a deterministic offline engine (no real branches/PRs). Safe for demos.
  • claude_code - the real Claude Code engineer. Needs the claude CLI on PATH, a writable BLOOM_ENGINEER_WORKSPACE_ROOT, and a configured GitHub App (to push + open PRs). GET /health reports engine.mode and a per-check engine.ready summary, so you can confirm the mode can actually run before and after a deploy. See engineering-engine.md.

Set the mode (and, for claude_code, the knobs above) as BLOOM_* values in the same shared app configuration as any other setting (step 3). Never enable claude_code without first checking engine.ready is true.

Switching the LLM provider to OpenClaw (M8)​

BLOOM_LLM_PROVIDER is one config value among the shared app configuration in step 3 above; this just spells out the OpenClaw-specific rollout. OpenClaw is a pluggable LLMProvider backend only - switching to it does not change who owns SDLC orchestration (Bloom's own workflow engine still does; see architecture.md).

Required repository/environment config (names match .env.example and the settings surface in apps/server/src/settings.ts):

NameKindRequiredNotes
BLOOM_LLM_PROVIDERvariableyesSet to openclaw to select this backend.
BLOOM_OPENCLAW_BASE_URLvariableyes, when BLOOM_LLM_PROVIDER=openclawThe gateway's URL. No safe default - startup fails fast if unset so production never silently falls back to a local gateway.
BLOOM_OPENCLAW_API_KEYsecretonly if the gateway requires authSent as Authorization: Bearer <key>.
BLOOM_LLM_MODELvariablenoThe model passed to the gateway (e.g. openclaw/default).
BLOOM_LLM_FALLBACK_PROVIDERvariablenoBuild-time safety net: if openclaw fails to build (e.g. missing base URL), build this provider instead of crashing startup.

Rollout steps:

  1. Set BLOOM_OPENCLAW_BASE_URL (and BLOOM_OPENCLAW_API_KEY if the gateway needs one) at the repository level (shared) or on the target dev/production environment (per-environment gateway) - same config layering as any other BLOOM_* value (step 3 above).

  2. Optionally set BLOOM_LLM_FALLBACK_PROVIDER (e.g. anthropic) so a build-time config error such as a missing base URL degrades to a working provider at startup instead of crash-looping. Runtime gateway request failures are still surfaced to callers; fallback is not a retry path.

  3. Set BLOOM_LLM_PROVIDER=openclaw and redeploy (push to main for dev, or run the production workflow manually).

  4. Confirm the deploy: docker compose logs bloom should show no BLOOM_LLM_PROVIDER=openclaw requires BLOOM_OPENCLAW_BASE_URL to be set error, and no llm_provider.fallback_engaged warning (that would mean the primary config was rejected and the fallback engaged instead).

  5. Optionally run the local live harness before rollout. This harness was part of the Python suite and retired with apps/api (#808) - check out git d69f789 to run it:

    cd apps/api
    BLOOM_OPENCLAW_LIVE_TEST=1 \
    BLOOM_OPENCLAW_BASE_URL=http://127.0.0.1:18789 \
    BLOOM_LLM_MODEL=openclaw/default \
    uv run pytest tests/integration/test_openclaw_provider_harness.py

OpenClaw orchestration pilot and rollback (M9)​

OpenClaw orchestration is separate from the OpenClaw LLM provider above. The default runtime is still Bloom's built-in runtime, and the M9 pilot is intentionally narrow:

BLOOM_ORCHESTRATION_RUNTIME=builtin

To enable the pilot for the read-only progress refresh path only:

BLOOM_ORCHESTRATION_RUNTIME=openclaw_pilot
BLOOM_OPENCLAW_ORCHESTRATION_PILOT_NODES=track_progress
BLOOM_OPENCLAW_ORCHESTRATION_FAILURE_MODE=fallback

Required repository/environment config (names match .env.example and the settings surface in apps/server/src/settings.ts):

NameKindRequiredNotes
BLOOM_ORCHESTRATION_RUNTIMEvariableyesbuiltin by default. Set openclaw_pilot only for an approved pilot rollout.
BLOOM_OPENCLAW_ORCHESTRATION_PILOT_NODESvariablenoComma-separated tool-node allowlist. Default is track_progress; keep it narrow.
BLOOM_OPENCLAW_ORCHESTRATION_PILOT_AGENT_NODESvariablenoComma-separated state-only agent-node allowlist (M10-5). Empty by default - no agent node is piloted out of the box. Set prd_authoring to opt in; it must also be in OpenClawGatewayNodeExecutor's own DEFAULT_NODE_AGENT_MAP.
BLOOM_OPENCLAW_ORCHESTRATION_FAILURE_MODEvariablenofallback by default. Use fail_closed only for deliberate drills or strict pilot validation. Applies to both allowlists above.
BLOOM_OPENCLAW_ORCHESTRATION_EXECUTOR_BASE_URLvariableyes, when BLOOM_ORCHESTRATION_RUNTIME=openclaw_pilot and BLOOM_OPENCLAW_ORCHESTRATION_FAILURE_MODE=fail_closedThe concrete orchestration-runtime executor's Gateway URL (M10-1). Separate from BLOOM_OPENCLAW_BASE_URL below - see Concrete orchestration executor config.
BLOOM_OPENCLAW_ORCHESTRATION_EXECUTOR_API_KEYsecretonly if the gateway requires authSent as Authorization: Bearer <key> by the executor. Never expose it to clients, logs, issue bodies, or PR comments.
BLOOM_OPENCLAW_ORCHESTRATION_EXECUTOR_TIMEOUT_SECONDSvariablenoPer-request timeout for the executor client (default 30).
BLOOM_OPENCLAW_BASE_URLvariableno for the current runtime pilotThis is the OpenClaw LLM-provider base URL (required only when BLOOM_LLM_PROVIDER=openclaw). It does not configure the orchestration-runtime executor above, even when pointed at the same gateway.
BLOOM_OPENCLAW_API_KEYsecretno for the current runtime pilotThe OpenClaw LLM-provider API key. Likewise does not satisfy the orchestration-executor config above. Never expose it to clients, logs, issue bodies, or PR comments.

Controls and limits:

  • BLOOM_ORCHESTRATION_RUNTIME=builtin disables the pilot and routes all workflow execution through Bloom's built-in runtime.
  • BLOOM_OPENCLAW_ORCHESTRATION_PILOT_NODES is a narrow tool-node allowlist. Keep it at track_progress until additional nodes are explicitly reviewed.
  • BLOOM_OPENCLAW_ORCHESTRATION_PILOT_AGENT_NODES is a separate, narrower agent-node allowlist (M10-5), empty by default. prd_authoring is the only reviewed state-only agent node - see docs/design/openclaw-workflow-mapping.md for why it was selected.
  • BLOOM_OPENCLAW_ORCHESTRATION_FAILURE_MODE=fallback runs the built-in node handler if the pilot executor fails. fail_closed raises an operator-visible error instead.
  • The pilot does not move webhook handling, durable job ownership, state storage, reducers, GitHub mutations, Telegram sends, PR review publication, merge decisions, or milestone closure into OpenClaw.
  • A centralized, code-owned capability policy (M10-6, bloom/workflow/openclaw_policy.py) is a third, independent gate above the two allowlists: it rejects any node that is not a tool/ agent node, is side-effecting, falls outside its own hard-coded node set, or declares writes outside what that node id is allowed to write. At the end of M10 exactly two node ids can ever route through OpenClaw, regardless of operator config: track_progress (tool) and prd_authoring (agent). Widening this set requires editing openclaw_policy.py and its tests, never a config change alone. See docs/design/openclaw-orchestration-runtime.md.

Monitor the active runtime via /health. The response reports the configured runtime, active runtime, pilot node allowlists, failure mode, concrete-executor availability, the capability-policy summary, and correlation-store health. It intentionally omits OpenClaw gateway URLs, API keys, prompts, and model responses.

Example health check:

curl -fsS http://127.0.0.1:8000/health | jq '.orchestration'

Expected default shape:

{
  "configured_runtime": "builtin",
  "active_runtime": "builtin",
  "openclaw_enabled": false,
  "pilot_nodes": ["track_progress"],
  "pilot_agent_nodes": [],
  "failure_mode": "fallback",
  "concrete_executor": {
    "configured": false,
    "available": false
  },
  "policy": null,
  "correlation_store": {
    "backend": "memory",
    "healthy": true
  }
}

policy is null for the built-in runtime (nothing to report) and, once the pilot is active, a secret-safe summary shaped like:

{
  "policy_allowed_tool_nodes": ["track_progress"],
  "policy_allowed_agent_nodes": ["prd_authoring"],
  "configured_tool_nodes": ["track_progress"],
  "configured_agent_nodes": [],
  "last_decision": null,
  "decision_counts": {}
}

Local validation before rollout:

Historical (Python-era) command blocks. The cd apps/api / uv run commands in the rest of this section and in the flip-readiness section below retired with the Python backend (#808); run them at git commit d69f789. The TS server ports the deterministic decision core (apps/server/src/workflow/openclawReadiness.ts, exercised by the regular pnpm run test) but does not wire the live flip-readiness CLI or the orchestration /health surface.

  1. Run the focused runtime and API checks:

    cd apps/api
    uv run pytest tests/unit/test_config.py tests/unit/test_runtime.py tests/unit/test_openclaw_policy.py tests/integration/test_api.py tests/integration/test_coordination_flow.py tests/integration/test_openclaw_readiness.py -q

  2. Run the flip-readiness command against the config you intend to deploy (see Flip-readiness check (M10-7) below) - this is the fast, in-process gate and should be green before you go any further:

    cd apps/api
    BLOOM_LLM_PROVIDER=fake \
    uv run python -m bloom.workflow.openclaw_readiness \
      --runtime openclaw_pilot \
      --pilot-nodes track_progress \
      --failure-mode fallback

  3. Start the API with the pilot flag and the deterministic fake LLM:

    cd apps/api
    BLOOM_LLM_PROVIDER=fake \
    BLOOM_ORCHESTRATION_RUNTIME=openclaw_pilot \
    BLOOM_OPENCLAW_ORCHESTRATION_PILOT_NODES=track_progress \
    BLOOM_OPENCLAW_ORCHESTRATION_FAILURE_MODE=fallback \
    uv run uvicorn bloom.api.app:create_app --factory

  4. In another shell, verify the runtime surface:

    curl -fsS http://127.0.0.1:8000/health | jq '.orchestration'

    Confirm configured_runtime=openclaw_pilot, active_runtime=openclaw_pilot, openclaw_enabled=true, pilot_nodes=["track_progress"], and correlation_store.healthy=true.

Staged rollout:

  1. Run the flip-readiness command (step 2 above) against the exact env vars you are about to set on the dev environment. Do not proceed if it reports NOT READY.

  2. Enable the pilot in dev with:

    BLOOM_ORCHESTRATION_RUNTIME=openclaw_pilot
    BLOOM_OPENCLAW_ORCHESTRATION_PILOT_NODES=track_progress
    BLOOM_OPENCLAW_ORCHESTRATION_FAILURE_MODE=fallback

  3. Redeploy dev by merging to main or running deploy-dev.yml.

  4. Check /health on the dev API and confirm the pilot values above.

  5. Watch API logs for secret-safe lifecycle events:

    docker compose logs -f bloom | grep -E 'openclaw_runtime|openclaw_policy|openclaw_executor'

    Expected event names include openclaw_runtime.node_started, openclaw_runtime.node_succeeded, openclaw_runtime.node_failed, openclaw_runtime.cached_result, and openclaw_policy.rejected (a policy-rejected node that was still present in the operator allowlist - should not appear unless config drifted from the M10 node set above).

  6. Keep fallback during the first dev soak so the built-in node handler remains the safety net if the pilot executor fails.

Production enablement:

  1. Confirm dev has passed local checks, CI, /health, and log review.
  2. Set the same runtime variables on the production environment only after approving the production rollout.
  3. Run the flip-readiness command one more time with --runtime openclaw_pilot and the exact variables about to be set on production (including --executor-base-url if set) - this is the last automated check before a real environment's config changes.
  4. Start production with BLOOM_OPENCLAW_ORCHESTRATION_FAILURE_MODE=fallback.
  5. Verify production /health reports the pilot values, capability-policy summary, and correlation-store health.
  6. Before switching to fail_closed, re-run the flip-readiness command with --failure-mode fail_closed and the production executor URL - Settings itself will refuse to boot with fail_closed and no executor URL configured, but the readiness command catches this (and the executor-availability/harness checks) before you touch the real environment. For openclaw_pilot + fail_closed specifically, the command now requires a real, reachable run of the live-gateway harness against that exact executor URL by default - /health reporting concrete_executor.available=True from wiring/config alone is not sufficient, so a fake or unreachable executor URL correctly reports NOT READY instead of a false READY. Do not run this step with --skip-live-gateway/--offline-only against production - that flag is an explicit, visibly-marked non-production simulation and never approves a fail_closed production flip; see Flip-readiness check (M10-7) below. Only flip to fail_closed for deliberate operator drills or once the pilot node's failure handling has been reviewed for the target environment.

Emergency rollback:

  1. Set BLOOM_ORCHESTRATION_RUNTIME=builtin.
  2. Redeploy or restart the API process.
  3. Confirm /health reports configured_runtime=builtin, active_runtime=builtin, and openclaw_enabled=false.

Rollback does not require a database migration. Existing runtime correlation rows can remain for audit; Bloom continues from persisted RunState through the built-in runtime.

Milestone completion criteria:

  • The built-in runtime remains implemented, tested, and documented as the default.
  • OpenClaw orchestration remains a pilot behind explicit config.
  • Removing the built-in runtime is not part of M9 and must be scheduled as a separate, explicitly-approved milestone if it ever becomes desirable.

M10 completion (concrete executor, agent-node pilot, capability policy, flip-readiness). At the end of M10:

  • A real, gateway-backed OpenClawGatewayNodeExecutor (not a fake) is wired into create_app.
  • The pilot covers exactly two nodes - track_progress (tool) and prd_authoring (agent) - enforced in code by the capability policy, independent of operator config.
  • Unsupported and side-effecting workflow nodes (human, event_wait, map, router, GitHub mutations, Telegram sends, deploy/CI triggers, PR review publication, merge decisions, milestone closure - see Exact M10 Routable Node Set) remain built-in-only; enabling any of them is out of scope for M10 and requires a deliberate, separately-reviewed change to openclaw_policy.py.
  • A machine-checkable flip-readiness command exists and is part of the rollout procedure above.
  • Next recommended expansion milestone (not started): pilot one additional read-only or advisory node - candidates noted in Pilot Path Recommendation include a status/monitoring answer path or a pre-publication specialist-review draft - through the same double-allowlist-plus-capability-policy pattern, only after this M10 pilot has run in production behind the flip-readiness check for a full soak period.

Flip-readiness check (M10-7)​

bloom.workflow.openclaw_readiness is a machine-checkable readiness gate operators run before switching an environment's BLOOM_ORCHESTRATION_RUNTIME to openclaw_pilot - and especially before BLOOM_OPENCLAW_ORCHESTRATION_FAILURE_MODE=fail_closed, where a misconfiguration is no longer silently absorbed by the built-in fallback. (Python-era CLI - retired with apps/api (#808), run it at git d69f789; the deterministic decision core is ported in apps/server/src/workflow/openclawReadiness.ts.)

cd apps/api
uv run python -m bloom.workflow.openclaw_readiness \
  --runtime openclaw_pilot \
  --failure-mode fail_closed \
  --pilot-nodes track_progress \
  --pilot-agent-nodes prd_authoring \
  --executor-base-url http://127.0.0.1:18789

The --runtime/--failure-mode/--pilot-nodes/--pilot-agent-nodes/--executor-base-url flags override the corresponding Settings field for this check only - they never write to .env or the process environment, so this is safe to run against a target environment's real configuration to preview a flip without changing it. Omit a flag to check whatever that field is currently set to (from the process environment / .env) - useful for verifying an already-live config post-deploy.

What it checks, all in-process (no separate server; a live OpenClaw gateway is only required for openclaw_pilot + fail_closed - see the live-gateway bullet below):

  • Config: Settings constructs without error (this alone exercises the existing fail_closed-without-executor-url guard), the configured pilot allowlists are a subset of the M10 capability policy's hard-coded node set, and (for fail_closed) the executor base URL is configured.
  • Health: GET /health (built via the real create_app, called in-process) reports status: ok.
  • Active runtime: orchestration.active_runtime == "openclaw_pilot" and openclaw_enabled == true.
  • Executor availability: orchestration.concrete_executor reports both configured and available. This reflects wiring/config only, not real reachability - it can be true even when the configured URL is fake or unreachable, which is exactly why the live-gateway check below exists and is required for fail_closed.
  • Correlation-store health: orchestration.correlation_store.healthy.
  • Policy wiring: the live runtime's capability-policy summary matches the configured allowlists (catches stale wiring, not just stale config).
  • The OpenClaw harness (offline): an offline (httpx.MockTransport) run of the track_progress execution harness (bloom.workflow.openclaw_track_progress_harness) through the real OpenClawPilotRuntime -> OpenClawGatewayNodeExecutor -> OpenClawExecutorGatewayClient code path - proving request shape, idempotency reuse, and Bloom's own reducer application still work, with no live gateway or network access. This check always runs and is always required.
  • The OpenClaw harness (live gateway): the same harness against the actual configured --executor-base-url (or BLOOM_OPENCLAW_ORCHESTRATION_EXECUTOR_BASE_URL) - the only check that proves the configured executor URL is genuinely reachable and executable, not just syntactically configured.
    • Required by default for openclaw_pilot + fail_closed. A fake or unreachable executor URL now correctly reports NOT READY; /health's concrete_executor.available alone is never sufficient for a fail_closed flip.
    • Optional everywhere else (fallback mode, or builtin runtime): opt in with BLOOM_OPENCLAW_EXECUTOR_LIVE_TEST=1 plus an executor base URL (mirrors tests/integration/test_openclaw_track_progress_harness.py's live check); otherwise it is skipped and marked [WARN]/optional in the output.
    • Explicit offline-only escape hatch: pass --skip-live-gateway (alias --offline-only) to skip the required live check for openclaw_pilot + fail_closed - for CI/docs dry runs only. This is always visibly marked in the report (offline_only_simulation: true in --json output) and in the final banner (READY (OFFLINE-ONLY SIMULATION) instead of plain READY). It does not approve a fail_closed production flip - never use it against the real production executor URL; re-run without it against the real gateway before flipping production.

The command prints one [OK]/[FAIL]/[WARN] line per check (add --json for a machine-readable report) and exits 0 only when every required check passes; exits 1 with a NOT READY message on stderr otherwise. No check output ever includes an API key or a gateway URL. Unit/integration coverage lives in tests/integration/test_openclaw_readiness.py, including dedicated tests asserting: no configured secret ever appears in the rendered report; a fake/unreachable executor URL is NOT READY by default under fail_closed; and --skip-live-gateway/--offline-only is visibly marked as a non-production simulation.

Concrete orchestration executor config (M10-1)​

The concrete gateway-backed executor has its own config surface and a defined contract (endpoint, request/response shape, idempotency, timeout, auth posture, unsupported-node behavior) in docs/design/openclaw-orchestration-runtime.md. create_app wires it when BLOOM_OPENCLAW_ORCHESTRATION_EXECUTOR_BASE_URL is configured, and falls back to UnavailableOpenClawNodeExecutor otherwise.

BLOOM_OPENCLAW_ORCHESTRATION_EXECUTOR_BASE_URL=
BLOOM_OPENCLAW_ORCHESTRATION_EXECUTOR_API_KEY=
BLOOM_OPENCLAW_ORCHESTRATION_EXECUTOR_TIMEOUT_SECONDS=30

This config is deliberately separate from BLOOM_OPENCLAW_BASE_URL/BLOOM_OPENCLAW_API_KEY (the OpenClaw LLM-provider config, BLOOM_LLM_PROVIDER=openclaw) - the two are validated and consumed independently, even when they happen to point at the same physical gateway.

Validation and health behavior:

  • Startup fails fast with an actionable error if BLOOM_ORCHESTRATION_RUNTIME=openclaw_pilot and BLOOM_OPENCLAW_ORCHESTRATION_FAILURE_MODE=fail_closed are set without BLOOM_OPENCLAW_ORCHESTRATION_EXECUTOR_BASE_URL:

    BLOOM_ORCHESTRATION_RUNTIME=openclaw_pilot with BLOOM_OPENCLAW_ORCHESTRATION_FAILURE_MODE=
    fail_closed requires BLOOM_OPENCLAW_ORCHESTRATION_EXECUTOR_BASE_URL to be set. ...

  • BLOOM_OPENCLAW_ORCHESTRATION_FAILURE_MODE=fallback never requires this config - it stays deployable with no OpenClaw executor present at all.

  • /health's orchestration.concrete_executor reports configured (whether BLOOM_OPENCLAW_ORCHESTRATION_EXECUTOR_BASE_URL is set) and available (whether the runtime's wired executor is actually usable). With no executor configured, both are false:

    curl -fsS http://127.0.0.1:8000/health | jq '.orchestration.concrete_executor'
    {
      "configured": false,
      "available": false
    }

GitHub App private key (file form)​

The App key is not part of .env; the bloom (and swarm) container mounts ./secrets read-only at /app/secrets, and BLOOM_GITHUB_APP_PRIVATE_KEY_PATH points at the in-container path. exe.dev's SSH server has no scp/sftp, so stream the .pem over an exec channel with cat (from your local machine, once):

ssh exedev@<vmname>.exe.xyz \
  'mkdir -p /home/exedev/bloom-ai/secrets \
   && cat > /home/exedev/bloom-ai/secrets/github-app.pem \
   && chmod 644 /home/exedev/bloom-ai/secrets/github-app.pem' \
  < your-app.private-key.pem
BLOOM_GITHUB_APP_PRIVATE_KEY_PATH=/app/secrets/github-app.pem   # in-container path

The deploy preserves secrets/ across syncs, so this is a one-time copy.

k3s dev runtime (M24 - parallel to compose until cutover)​

M24 stands up a single-node k3s cluster on the same dev VM as the future runtime for dev. It runs alongside the compose deploy above - compose stays the live path (exe.dev edge -> 8080) until the M24-5 cutover; k3s owns 6443 (API), 10250 (kubelet), and 80/443 (the built-in Traefik ingress, kept enabled - decision + rationale in deploy/k8s/README.md). No ports overlap.

  • Bootstrap / upgrade (idempotent; pin with K3S_VERSION, or K3S_CHANNEL, default stable):

    ssh exedev@bloom-server.exe.xyz 'bash -s' < deploy/k8s/bootstrap-k3s.sh

    Installs/upgrades k3s, waits for the node Ready + core system deployments, writes ~/.kube/config for the login user, and asserts the bloom namespace + ghcr-pull imagePullSecret. bootstrap-k3s.sh status re-checks health; bootstrap-k3s.sh ghcr-secret refreshes just the pull secret.

  • GHCR pulls: the private API image pulls through the bloom/ghcr-pull imagePullSecret. GHCR rejects GitHub App installation tokens, and the VM keeps no long-lived registry credential (same posture as the compose deploy) - so CI refreshes the secret with its ephemeral job token at deploy time (M24-5), and app pods use imagePullPolicy: IfNotPresent (M24-2) so restarts reuse the local image, exactly like compose restarts do today.

  • Teardown (removes the cluster and all its state; compose unaffected):

    ssh exedev@bloom-server.exe.xyz 'sudo /usr/local/bin/k3s-uninstall.sh'

  • App manifests (M24-2): the bloom Helm chart under deploy/helm/bloom is the single source of truth for the API, web, and (guarded, off by default) swarm-simulator workloads - probes on the API's real /health, config via ConfigMap, secret refs left optional until M24-4, and the image tag pinned per deploy to the commit SHA (--set image.tag=<sha>), which is what keeps helm rollback equivalent to the M15 image rollback. CI lints, renders, and unit-tests the chart on every PR.

  • Env overlays + ingress/TLS (M24-3): one overlay per environment (values-dev/-staging/-prod.yaml) so promotion is the same helm upgrade with the next values file - never a manifest rewrite. Dev rides Traefik behind the exe.dev edge (which terminates TLS); staging/prod terminate TLS in-cluster via templated cert-manager Issuer/Certificates. What changes per env vs. what stays identical: the "Environment overlays & promotion" section of the chart README.

  • Platform provisioning as code (M24-8): the OpenTofu module under deploy/tofu provisions the staging/prod substrate the overlays above assume - DOKS cluster + node pool, DNS, ingress-nginx + cert-manager bootstrap, and the Neon preview project - one module, per-env tfvars ("staging looks like prod"). Authored + offline-validated only ($0): CI runs tofu fmt/init -backend=false/validate on every PR; plan/apply and the remote state store are M25-1 (#400).

CI deploy via Helm + rollback (M24-5)​

deploy-k3s-dev.yml deploys main to the k3s cluster on the same trigger as the compose deploy (CI success on main, plus manual dispatch) and shares the deploy-dev concurrency lock, so the two deploys and the secret sync always serialize. Until the cutover below, both deploys run on every merge - compose stays the live path behind the exe.dev edge while the helm deploy proves parity. Each run:

  1. syncs the sealed secrets (scripts/deploy/k8s-secrets-dev.sh, M24-4), so pods always roll out with the current Actions secrets;
  2. runs the helm target scripts/deploy/helm-dev.sh, which refreshes the ghcr-pull imagePullSecret with the workflow's ephemeral GITHUB_TOKEN, ensures the pinned + checksum-verified helm CLI on the VM (deploy/k8s/helm-cli.sh), streams the chart over the SSH exec channel, and runs helm upgrade --install bloom -n bloom -f values-dev.yaml --set image.tag=sha-<12> --wait;
  3. health-gates success twice: --wait holds until the new pods pass their readiness probes (the API's real GET /health), then an endpoint check curls /health through Traefik with the dev ingress host - the exact request the exe.dev edge forwards after the cutover. Either failing turns the run red with pod/log diagnostics; nothing is recorded as deployed.

The SHA-pinned tag keeps the M15 immutable-deploy semantics: helm's in-cluster revision history replaces the compose path's .deployed_tag/.deployed_tag_prev files. Rollback is the Rollback (dev, k3s) workflow (deploy-rollback-k3s-dev.yml): one click runs helm rollback to the previous revision (or a chosen one via the revision input), through the same probe + endpoint health gate. On the VM, helm history bloom -n bloom lists revisions and helm get values bloom -n bloom --revision <n> shows the SHA a revision pinned.

The web workload stays --set web.enabled=false in this deploy: no ghcr.io/hidden-claw/bloom-ai-web image is published yet, and dev web keeps shipping to Cloudflare Pages (deploy-web-dev.yml) - unchanged by the cutover below.

Post-deploy e2e on dev/staging: dormant wiring + activation (M24-9)​

After a dev deploy, E2E (dev) (e2e.yml) runs the full Playwright suite (apps/web/e2e) against the deployed dashboard; E2E (staging) (e2e-staging.yml) is its manual-dispatch staging twin until M25 ships a staging deploy workflow to hook. Both are thin callers of the reusable E2E (post-deploy) stage (e2e-post-deploy.yml), which authenticates through the password-gated test-login path and hands the verdict to scripts/deploy/e2e-promote-rollback.sh (E2E-7):

  • pass → promote the running dev image to "last e2e-green" (the rollback target);
  • fail after Deploy (dev) → compose rollback to the last e2e-green image;
  • fail after Deploy (dev, k3s) → helm rollback to the previous revision;
  • anything else (web deploy, manual dispatch, unarmed env, staging until M25) → clean no-op.

The whole decision matrix is unit-tested offline (stubbed ssh, no live deploy): scripts/deploy/tests/test_e2e_promote_rollback.sh, run by CI's deploy-scripts gate and make e2e-verdict-test.

Everything ships dormant. With no E2E_BASE_URL / E2E_STAGING_BASE_URL repository variable set, the workflows skip - and the committed chart overlays keep test-login off by omission (values-dev.yaml), so merging code can never silently expose the test-login form on a public site.

Activation runbook (operator, per environment) - dev shown; staging is identical with the staging environment and E2E_STAGING_BASE_URL:

  1. Arm the API's test surface. Set two Actions secrets in the dev environment (Settings → Environments → dev): BLOOM_ENABLE_TEST_LOGIN=true and a strong random BLOOM_TEST_LOGIN_PASSWORD (e.g. openssl rand -base64 30). Run Sync secrets (dev) and restart the pods (kubectl -n bloom rollout restart deploy, or re-run the deploy workflows - pre-cutover, the live compose path needs a Deploy (dev) run to re-render its .env). Key-delivery details: deploy/k8s/README.md.
  2. Verify the gate: curl -s https://bloom-server.exe.xyz/api/auth/config returns "testLogin": true and "testLoginRequiresPassword": true.
  3. Point the suite at the env. Set the repository variable E2E_BASE_URL to the deployed origin (https://bloom-server.exe.xyz for dev); staging uses E2E_STAGING_BASE_URL. Repo-level variables, deliberately: a job-level if cannot read environment-scoped variables, so the dormant gate keys off repo variables.
  4. Dispatch one run of E2E (dev) and expect green - including the "Promote image to last e2e-green" step recording the running tag. From then on every dev deploy triggers the suite automatically (staging stays dispatch-only until M25).

Security note - the visible-but-locked test-user form. Enabling test-login makes a "test user" sign-in form visible on that environment's login page (login.tsx). It is locked behind the password (X-Bloom-Test-Login-Password gates test-login and every /api/testing/* route), and it is structurally impossible in production: config.py::_forbid_test_login_in_production refuses to construct a production config with it enabled (prod gets a separate unauthenticated smoke instead, #424). Showing the form on dev/staging is the deliberate ops decision the activation makes. Disarm by unsetting the repo variable (the suite goes dormant again) and deleting the two environment secrets, then re-running the sync + a rollout (the form disappears).

Prod post-deploy smoke: unauthenticated checks + security guard (M24-10)​

Production cannot run the authenticated E2E lane above - test-login is structurally forbidden there - so Smoke (prod) (smoke-prod.yml) is prod's post-deploy verdict. It fires after Deploy (production) / Deploy Web (production) completes successfully (plus manual dispatch), with the same non-blocking posture as E2E (dev): a failure is a red run + step summary, never a block on the deploy that already happened. The checks (scripts/deploy/smoke-prod.sh) use only public, unauthenticated surface - no credentials, no secrets, no environment: binding:

  • GET /health → 200 with the version + engine readiness fields;
  • GET /api/auth/config → 200 with testLogin:false - the security regression guard: a build that ever ships prod with the test-login bypass reachable fails loudly here (it fails closed - a missing key or unreadable body also reds the run);
  • GET / → 200 with the login page shell (the SPA mount <div id="root"> + Bloom title);
  • every /assets/*.js|css bundle the shell references → 200 (the render preconditions).

All checks run (no fail-fast) so one red run shows the whole picture. The matrix is unit-tested offline against a stub curl: scripts/deploy/tests/test_smoke_prod.sh, run by CI's deploy-scripts gate and make smoke-prod-test.

Rollback on a red smoke is manual, by design - the deliberate opposite of dev's auto-rollback-to-last-e2e-green: prod deploys are manual dispatches with a human already at the wheel (dev auto-deploys on every merge, so nobody is watching); prod keeps no "last e2e-green" bookkeeping (only the dev e2e lane advances it, and scripts/deploy/e2e-promote-rollback.sh rejects prod outright); and an unauthenticated smoke is too shallow a signal to auto-revert production on - a transient edge/CDN blip must not flap the prod release. A red run's summary prints the manual path: Deploy (production) with mode: rollback for the API, and the Cloudflare Pages deployment list (or a re-run of Deploy Web (production) from the last good commit) for web.

Ships dormant until M25 stands prod up. Activation (operator):

  1. Set the repository variable SMOKE_PROD_BASE_URL to the prod dashboard origin (repo-level deliberately - a job-level if cannot read environment-scoped variables).
  2. While prod web is Cloudflare Pages, also set SMOKE_PROD_API_BASE_URL to the prod API origin: the Pages proxy (apps/web/functions) forwards /api/* only, and /health is not under /api. A single-origin prod (an ingress routing /health like dev's k3s ingress) needs only the one variable.
  3. Dispatch one Smoke (prod) run and expect green. From then on every prod deploy triggers it automatically.

An authenticated prod check (a synthetic real-OAuth account or signed monitoring token) is a deliberate follow-up, not part of this lane.

Cutover runbook: compose -> k3s (the documented switch)​

The compose deploy is kept until k3s parity is verified, then retired:

  1. Parity soak. Over a few merges to main, both deploy workflows stay green. On the VM, the two runtimes answer identically:

    curl -fsS http://127.0.0.1:8080/health                                  # compose
    curl -fsS -H 'Host: bloom-server.exe.xyz' http://127.0.0.1:80/health    # k3s via Traefik

  2. Flip the exe.dev edge from compose (8080) to Traefik (80):

    ssh exe.dev share port bloom-server 80

  3. Verify https://bloom-server.exe.xyz/health is served by k3s (kubectl -n bloom logs deploy/bloom-api shows the requests; compose logs stay quiet). Telegram/GitHub webhooks and the Pages /api proxy follow automatically - same hostname, and the dev ingress routes /api, /webhooks, /health, /metrics to the API.

  4. Retire the compose path (a follow-up PR + VM cleanup once the flip has soaked): remove the workflow_run trigger from deploy-dev.yml (keeping workflow_dispatch as a temporary escape hatch, then delete the workflow together with deploy-rollback-dev.yml), retire the compose branches of the post-deploy e2e verdict handler (scripts/deploy/e2e-promote-rollback.sh - the Deploy (dev, k3s) trigger already rolls back via helm (M24-9), but promote and the Deploy (dev) rollback still act on the compose stack's bookkeeping), and stop the stack on the VM: cd ~/bloom-ai && docker compose down.

  5. Undo (if needed). The flip is one command to reverse - ssh exe.dev share port bloom-server 8080 restores compose as the live path (do not stop the compose stack until the soak is over).

Until step 4 lands, the compose deploy is deprecated but live; after it, the helm deploy is the only dev path (production/staging follow in M25).

Admin dashboard: production posture (M27)​

The admin console is one codebase with a deploy-time mode switch, not a second service (design 3.3). BLOOM_ADMIN_API_MODE decides what an API instance serves:

ModeServesUse
disabledUser surface only; /api/admin/* is never mounted (404, indistinguishable from absent)The user-facing production API
onlyAdmin surface only (+ /health, /metrics); user/webhook routers absent; scheduler + job-runner never startThe admin production API instance
mounted (default)Both surfaces on one instanceDev / combined deployments

The two-origin model. The user dashboard and the admin console are different registrable origins (bloom-web vs bloom-admin Pages projects; deploy-admin-dev.yml / deploy-admin-staging.yml publish the latter). Each SPA reverse-proxies /api/* first-party through its own Pages Function, so each origin holds exactly one host-only session cookie: bloom_admin_session (Secure, HttpOnly, SameSite=Lax, 12h TTL, never a Domain attribute) exists only on the admin origin, and nothing in the user origin's cookie jar or storage authorizes anything on the admin surface - and vice versa. The admin origin is deliberately absent from the API's CORS allowlist; it never needs an entry because the proxy keeps it same-origin.

Production checklist (admin instance, BLOOM_ADMIN_API_MODE=only):

  • BLOOM_ADMIN_DASHBOARD_URL - the admin origin; the OAuth callback, the access-denied redirect, and the origin-check allowlist all derive from it. Register <origin>/api/admin/auth/google/callback in the Google OAuth client alongside the user one.
  • BLOOM_ADMIN_EMAILS - bootstrap superadmin seeding (idempotent; thereafter grants are managed in the UI and audited).
  • Hardening knobs (M27-14, on by default whenever the admin surface is mounted): the origin-check middleware 403s any state-changing /api/admin request whose Origin/Referer is not the admin origin (extras via BLOOM_ADMIN_ALLOWED_ORIGINS, normally unset) and refuses non-JSON mutation bodies; /api/admin/auth/* is rate limited per client IP (BLOOM_ADMIN_AUTH_RATE_LIMIT_REQUESTS/..._WINDOW_SECONDS, default 20 per 60s). Run uvicorn with proxy headers enabled behind the edge so the limiter keys on real client IPs, not the proxy's.
  • Forbidden in production by boot-time validators (Settings refuses to construct): BLOOM_ENABLE_TEST_LOGIN, BLOOM_ADMIN_TEST_GRANT, BLOOM_ADMIN_TEST_OIDC. These are the E2E-only surfaces; a misconfigured production deploy fails fast instead of running them.
  • The user-facing API instance sets BLOOM_ADMIN_API_MODE=disabled and needs none of the admin variables.

CSP / response headers. Every response from the admin origin - static assets via apps/admin/public/_headers, proxied /api responses via the Pages Function - carries the strict header set defined once in apps/admin/security-headers.ts (each directive's justification is documented there; apps/admin/src/test/csp-parity.test.ts fails CI if the two surfaces drift).

Threat-model verification: every §4.4 STRIDE row's mitigation is pinned to a test or documented check in docs/security/admin-threat-model.md.

Cloudflare Access in front of the admin origin (edge second factor)​

Al's M27-14 decision: the admin origin gets Cloudflare Access as an edge-level second factor - an attacker (or a stolen Google session) must also pass Access before a single byte of the admin SPA or its /api proxy is served. The free Zero Trust tier covers this team size.

Status: configured-as-documentation, not yet enforced. The admin origin (bloom-admin.pages.dev) first deploys after the m27 -> main merge, and Access rides the owner's Cloudflare Zero Trust account - so live enforcement is the operator's post-deploy step below. Nothing in CI asserts Access yet; do not treat this section as evidence of enforcement.

One-time prerequisite (free tier): Zero Trust setup on the Cloudflare account that owns the Pages projects - pick a team domain (<team>.cloudflareaccess.com). The default One-time PIN login method is sufficient (the allow-list below pins emails; the PIN proves control of the inbox); adding Google as an Access identity provider is optional polish.

Dev + staging (*.pages.dev origins) - dashboard procedure (the Pages-integrated path Cloudflare supports for pages.dev hostnames):

  1. Cloudflare dashboard -> Workers & Pages -> bloom-admin (dev; repeat for the staging project) -> Settings -> Access policy -> Enable. This creates a Zero Trust Access application protecting the project's preview deployments (*.bloom-admin.pages.dev).
  2. In Zero Trust -> Access -> Applications, edit that application and add the production hostname bloom-admin.pages.dev to its domains, so previews AND the live origin are both gated.
  3. Edit the application's policy: action Allow, include Emails = the same list as BLOOM_ADMIN_EMAILS (plus any granted operators/viewers). Session duration: 12h or less, never longer than the admin cookie TTL - Access must expire first or with it.

Production custom domain (admin.<domain>, once one exists in the account's zone) - config-as-code via the API (reproducible; also works for staging on a custom domain):

# ACCOUNT_ID + a token with Access: Apps and Policies Edit. Re-runnable: POST once, then
# manage by id (GET /accounts/$ACCOUNT_ID/access/apps lists existing apps).
curl "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/access/apps" \
  --request POST \
  --header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  --json '{
    "type": "self_hosted",
    "name": "Bloom Admin (production)",
    "domain": "admin.example.com",
    "session_duration": "12h",
    "policies": [
      {
        "name": "Bloom admins",
        "decision": "allow",
        "include": [
          { "email": { "email": "al@example.com" } }
        ]
      }
    ]
  }'

Keep the include email list in lockstep with the grant table's superadmin/operator set; Access is the outer gate, the in-app bloom_admin_grants check stays authoritative for roles.

Post-deploy verification (the operator's step, after m27 -> main deploys the admin origin):

  1. curl -sI https://bloom-admin.pages.dev/ - expect a 302 to https://<team>.cloudflareaccess.com/... (the Access login), not the SPA. An in-private browser must hit the Access gate before the Bloom admin login page.
  2. Complete Access with an allow-listed email, then the normal admin Google sign-in; confirm /api/admin/me succeeds end-to-end through both gates.
  3. Confirm a non-allow-listed email is refused by Access (never reaching the origin at all).
  4. Run the admin E2E suite against the deployed dev admin (E2E_BASE_URL=https://<dev-admin-origin> pnpm run test:e2e from apps/admin; see apps/admin/e2e/README.md for the flags the target API needs, and note the deployed run requires an Access service token or a temporary bypass policy for the suite's traffic - CI's per-PR runs use the local stack precisely so they need neither).

Operations​

  • Deploy: push to main (auto-deploys dev once CI is green), or run the workflow from the Actions tab. The deploy pulls the CI-built image by its commit-SHA tag. Until the M24-5 cutover this fires both dev deploys - compose (live) and k3s/Helm (parity) - see the cutover runbook above. Production is manual (workflow_dispatch).
  • Rollback: run Rollback (dev) from the Actions tab to redeploy the previous image on the compose path (.deployed_tag_prev), or Rollback (dev, k3s) for helm rollback on the k3s path; production uses Deploy (production) with mode: rollback. All are health-gated and reversible. (Rolling the app back does not roll back the database - see the data-rollback note above.)
  • Restart / logs (on the VM): docker compose restart · docker compose logs -f bloom
  • Auto-recovery: restart: unless-stopped restarts a crashed container; with the Docker service enabled, the stack also comes back after a reboot.

Local run​

cd apps/server && pnpm run dev   # the TS server with autoreload (or `make run` at the repo root)

To exercise the production image locally, build it from the repo root (docker build -f apps/server/Dockerfile .); the dev/prod deploys never build - they pull the CI-built image by tag. docker-compose.yml no longer defines workloads (the compose stack retired with the k3s cutover, #807). See also coordination.md for the retired engineer-swarm simulator.