Engineering engine

Bloom does not only coordinate engineering work as a Product Owner - it can execute it. Executing a delegated ticket happens behind the engineer-engine seam: a single, typed contract that both the production engine and an offline simulator satisfy, selected by configuration. The coordination, label, and review loop never depends on which engine is running.

This document describes the contract (milestone M6-1). The concrete real-work adapters - isolated workspace, Claude Code engineer, branch push and PR creation, the rework loop - land in later M6 tickets and plug into this same seam.

The seam

The atomic unit of work is exactly one GitHub issue. An engine takes an EngineRequest, does the work however it likes, and returns an EngineResult:

class EngineerEngine(Protocol):
    mode: str
    async def implement(self, request: EngineRequest) -> EngineResult: ...

Credentials, working directories, and command runners are the concrete engine's own constructor dependencies - never fields on the request or result. By construction the request and result carry no secrets, so a run is safe to persist and to log verbatim.

Request

EngineRequest carries everything needed to implement one issue: repo, base_branch, issue_number, issue_title, issue_body, the assigned engineer (eng:<name>), an optional explicit target_branch (otherwise derived as feat/issue-<n>, the one shared convention), the attempt number, and - for a rework - the review_feedback to address. A rework (attempt > 1) is required to carry feedback.

Result

EngineResult reports a typed outcome plus branch, pull_request_number / pull_request_url, a human-readable summary, redacted logs, and a failure_reason. Validation enforces the invariants: an opened/pushed result must name a branch and a PR; a blocked/failed result must give a reason.

EngineOutcome	Meaning	Drives ticket to
pr_opened	First implementation landed a branch + PR	status:in-review
changes_pushed	Rework pushed to the existing PR after a review	status:in-review
blocked	Needs clarification; ticket returns to the pool	status:blocked
failed	Engine ran but could not complete the work	status:needs-attention
skipped	Engine disabled/declined; no work attempted	(unchanged)

The outcome→label mapping lives in status_label_for(...) so the engine contract ties into the shared label vocabulary in exactly one place.

Run lifecycle

EngineRun is the durable, secret-free projection of one ticket's engineering run - enough to audit what happened and to resume after a restart. Its EngineRunState machine:

ready ─▶ in_progress ─▶ pr_opened ─▶ done
                            ▲   │
                            │   ▼
                     changes_requested         (review asks for changes; a rework returns to pr_opened)
   in_progress ─▶ blocked                      (returned to the pool for re-routing)
   in_progress ─▶ failed ─▶ needs_attention    (execution fault; escalated to a human)

Engine results drive the engine-owned transitions (EngineRun.with_result); the review-driven transitions (changes_requested, done) are applied by the orchestrator's review loop, so an engine result never lands the ticket by itself.

Workspace and branch isolation

A real engine run happens in an isolated git worktree off a shared base clone, managed by WorkspaceManager. Each issue is implemented on exactly one branch, feat/issue-<number>, cut from the default branch:

• Isolation. A worktree shares the base's object store but has its own working tree checked out from a committed ref - so a dirty base working tree never leaks into a run. A dirty base is refused by default; isolate_dirty_base opts into running anyway (the worktree isolates the uncommitted changes away).
• No duplication. If the issue already has a local branch (or an open PR, via an injected check), the run resumes that branch instead of creating a second one.
• Safe teardown. cleanup() removes the worktree after success or failure and keeps the branch (it holds the work); preparation is idempotent, resetting any leftover worktree first.

Subprocess access goes through an injectable git-runner seam, so the decision logic is unit-tested with no real git while integration tests drive real git against a temporary repository. The manager does not fetch, push, or open PRs - the network side is the publisher's job (below).

feat/issue-<n> is the one branch convention, defined once in the contract (bloom.engine.base.issue_branch) and shared by the request model, the workspace manager, and the publisher. The offline simulator reports the same convention but never touches git. A rework overrides it with the open PR's actual head branch (target_branch), whatever that is.

The Claude Code engineer

The production engine, ClaudeCodeEngine, composes the workspace with a Claude Code run to implement one issue:

1. Prepare an isolated workspace (above).
2. Brief - build_brief turns the issue into a strict, one-issue implementation prompt that bakes in the standards a good PR is judged against: prior-art search before building anything non-trivial, repo-standard tests, Conventional Commits, and no scope beyond the issue. A rework leads with the review feedback to address. The brief also carries the engineer's injected skillset (M22-5): a deterministic, provenance-tagged block of vendored knowledge skills selected from the skill registry - engineer_skill_block resolves the code-declared baseline plus the runner's add-only BLOOM_RUNNER_ENGINEER_SKILLS additions through the same capability-gated path the in-process agents use.
3. Run Claude Code through an injectable command seam (CommandRunner), so tests script success, non-zero exit, timeout, and malformed output with no real Claude session. The default runner captures stdout/stderr, enforces a timeout (killing the process), and reports a missing binary softly.
4. Verify the run actually committed work - HEAD is snapshotted before and after the run, and an unmoved HEAD is a failure. The engine trusts git state, not the model's prose, and the comparison also catches a rework that committed nothing (where counting commits beyond the base would wrongly count the branch's earlier ones).
5. Publish via the PullRequestPublisher seam, then return a typed result. The worktree is always torn down.

Every stored log and failure summary is passed through redact, which masks token-like strings (GitHub / Anthropic tokens, KEY=secret, Bearer …), and logs are bounded to their tail. Requests and results carry no credentials, so runs are safe to persist and surface.

Publishing: one issue = one branch = one PR

A completed run is published by GitHubPRPublisher, which enforces a hard invariant - one issue = one branch = one PR:

• Push first, always. On a rework the local branch carries new commits the open PR does not have yet, and pushing an already-up-to-date branch is a cheap no-op - so the push is unconditional and only PR creation is deduplicated. The default TokenBranchPusher pushes over HTTPS with a short-lived installation token handed to git via the environment (a credential helper reads it at runtime) - never via argv (readable by any ps on the host), the remote URL, or disk - reusing the existing GitHub App auth, with terminal prompts disabled so a broken credential fails fast.
• Dedupe - if an open PR already exists for the branch, it is reused, not duplicated.
• Open exactly one PR against the default branch with a reviewable body (what & why, evidence, and exactly one Fixes <repo>#<issue> line). Because the PR targets the default branch, that closing keyword actually closes the issue on merge. build_pr_body validates its own output, so a body that would close several issues can never be emitted.
• Label - moves only that one issue to status:in-review; no other issue's labels are touched.

Everything is behind seams (a narrow GitHub port + the pusher), so the whole path is tested with fakes - push/open success, duplicate prevention, in-review labelling, and the multi-issue-close regression - with no real remote.

Failure handling and escalation

Engine failures are classified, bounded, and made observable rather than silent or endlessly retried (failures.ts, engineRunner.ts):

• Classification - each failure maps to a FailureKind (setup, implementation, test, push, pr_creation, timeout, usage_limit). Infrastructure faults (setup, push, timeout, usage-limit) are retryable; a failure that reflects the work itself (implementation, test) is not
  • retrying it identically would just fail again.
• Bounded retry - the EngineRunner retries retryable failures with exponential backoff up to RetryPolicy.max_attempts; a usage-limit failure is a pause (waiting on a quota, not a bug).
• Escalation - a non-retryable failure, or one that keeps failing past the limit, moves the ticket to status:needs-attention with a useful GitHub comment and a concise owner notification.
• Observability - engine_run_started / retried / paused / failed / escalated events are emitted for the dashboard, and a secret-free run record (state, attempt, retries, failure kind/reason, last-attempt time) is persisted in durable workflow state under engine_runs, keyed by issue.

Everything derived from an exception is passed through redact before it is stored or surfaced.

Configuration

One engine is built per process, selected by build_engineer_engine(settings):

BLOOM_ENGINEER_ENGINE	Engine
disabled (default)	Bloom does not implement; an external swarm or humans pick up work
simulator	Deterministic in-process engine; no real branches or PRs (safe)
claude_code	Real Claude Code engineer on the remote runner VM (opt-in)

BLOOM_ENGINEER_EMAIL_DOMAIN sets the git-author domain commits are attributed to; BLOOM_ENGINEER_MAX_ATTEMPTS bounds automatic retries and BLOOM_ENGINEER_RUN_TIMEOUT_SECONDS the per-run budget. claude_code additionally needs the runner coordinates - BLOOM_ENGINE_RUNNER_URL and BLOOM_ENGINE_RUNNER_TOKEN - since the engine itself lives on the worker VM (below); the execution-local knobs (BLOOM_RUNNER_CLAUDE_BIN, BLOOM_RUNNER_WORKSPACE_ROOT, BLOOM_RUNNER_ENGINEER_SKILLS, ...) are the runner's own configuration.

Safeguards and readiness

Enabling production execution is always an explicit opt-in - the default is disabled. Selecting claude_code has real prerequisites on both sides. Bloom-side, validated by evaluate_engine_readiness: the runner URL, the service token, and a configured GitHub App (to mint per-run installation tokens). Runner-side, its own /health reports the claude binary, the subscription token, a writable workspace, git, and that ANTHROPIC_API_KEY is absent. GET /health on Bloom reports the configured engine.mode, the Bloom-side checks, and a live engine.runner probe, folding both into engine.ready (names + booleans only, never a secret). disabled and simulator have no external prerequisites.

disabled (the default) reproduces Bloom's pre-M6 behaviour: it delegates via labels and an external swarm harness or human engineers do the work - so turning on in-process execution is always an explicit opt-in and existing deployments are unaffected. simulator is the safe active mode: it fabricates a well-formed result with a synthetic PR reference and makes no GitHub calls, so offline demos exercise the full loop without doing real work. claude_code is the real engineer.

When an engine is active (not disabled), Bloom reworks a change-requested PR by invoking the engine on the same branch/PR, instead of waiting for an external actor to poll. With a job queue configured (production), the rework runs as a durable job - a real engine run can take tens of minutes and never runs inside the PR-event handler, where it would hold the project's thread lock; without a queue (dev/tests with the instant simulator) it runs inline. review_max_rounds still bounds the loop: after the limit Bloom stops reworking, marks the ticket status:needs-attention, comments on the PR, and notifies the owner. An engine that reports blocked is not escalated: the reason is surfaced, the ticket is unassigned and flagged status:blocked, and the reconcile re-routes it.

Choosing claude_code without the runner coordinates or a GitHub App fails fast with a clear configuration error rather than silently doing nothing; the app logs why and runs with no in-process engine (external-actor behavior).

Where the engine runs: the runner VM (M17)

Claude Code executes generated code - it installs dependencies and runs tests. That must not happen beside Bloom's GitHub App key, database DSN, and session secrets, so execution lives on a dedicated worker VM and Bloom talks to it over the owner's tailnet:

Bloom (API container)                    bloom-engine (worker VM)
  coordinator delegates a ticket
    -> engine_implement job (durable)
    -> RemoteEngine  --------- tailnet ---> engine-runner service
       + service bearer token                 POST /implement (single-flight)
       + short-lived GH installation token       bootstrap base checkout (M17-1)
                                                 worktree feat/issue-<n>
                                                 Claude Code (subscription auth)
                                                 push + open/reuse the PR
    <---------------- EngineResult ----------
  review loop, labels, engine_runs record

• Blast radius. The worker holds only the Claude subscription OAuth token and the short-lived per-run GitHub token from the request. Bloom's long-lived credentials never leave its container.
• Single-flight. The runner serves one run at a time; a busy runner answers 429 + Retry-After, which Bloom's classifier treats as a retryable pause.
• Billing. The Claude subprocess environment carries CLAUDE_CODE_OAUTH_TOKEN and is scrubbed of ANTHROPIC_API_KEY, so a run bills the subscription and can never silently bill the API. The runner's /health asserts the key's absence as a check.
• Exposure. The service binds the VM's tailscale address on a non-edge port: reachable from tailnet peers, never from the public internet.
• Observability. GET /metrics serves the runner's own Prometheus registry, bearer-authed with the same service token as /health (#355); see docs/observability.md.

Deploy or update it with deploy/engine-runner/deploy.sh [host] (tar-over-ssh, uv sync, a 0600 systemd EnvironmentFile created once, unit install + restart). The service token is generated on first deploy; put the same value in Bloom's BLOOM_ENGINE_RUNNER_TOKEN.

The runner is the one deliberate exception to the M24 Kubernetes-native deploy model: it stays on its dedicated VM (ADR 0001) because it executes untrusted generated code, and the k3s cluster shares a kernel with Bloom's secrets. With the app itself running as k8s pods, tailnet reachability is preserved by two standard mechanisms: pod egress to tailscale IPs is masqueraded through the node's tailscaled, and the *.ts.net MagicDNS names are pinned per environment via the chart's api.hostAliases (the analog of docker-compose's extra_hosts) - details and verification steps in the ADR.

Operating a live run

1. Confirm readiness: GET /health on Bloom shows engine.mode=claude_code, engine.ready=true, and an engine.runner.reachable=true probe.
2. Watch a run: journalctl -u bloom-engine-runner -f on the worker; Bloom emits engine_run_started / retried / paused / failed / escalated events for the dashboard.
3. Audit after the fact: the durable engine_runs record (per issue) holds state, attempt, retries, failure kind/reason, and last-attempt time - secret-free by construction.
4. Turn it off: set BLOOM_ENGINEER_ENGINE=disabled and redeploy; delegated tickets revert to external-actor behavior with no code change.