Observability: structured logging, correlation IDs & metrics

Structured logging + correlation implemented in M21 (#353); the metrics collection layer in #354; the /metrics scrape endpoints in #355; the alerting hooks on top of them in #356.

Bloom's backend logs are structured (structlog): every record is a stable machine-parseable event name plus key-value fields - JSON in production (json_logs=True), pretty console output in development. The goal is that "what happened on run X?" is answerable from the server side with one filter, which the M17 pilot showed plain text lines cannot do.

Correlation IDs

Every log record emitted inside a run carries correlation ids, merged in automatically from context (contextvars, via structlog.contextvars) - call sites never pass them manually, and concurrent runs are isolated because each asyncio task carries its own context copy.

Field	Meaning	Set where
run_id	One triggered unit of work, end to end: webhook/chat/tick -> queued job -> orchestrator -> engine. 16-hex, minted by bloom.logging.new_run_id().	Stamped into the job payload at enqueue (JobQueue.enqueue); inherited from the ambient run when a job is enqueued mid-run (e.g. the orchestrator scheduling an engine implement), so chained jobs stay on the run that caused them. Minted directly at queue-less entry points (web chat, credential updates) via ensure_run_id().
thread_id	The project the work belongs to.	From the job payload where it carries one; a chat job only knows its chat, so the orchestrator binds the thread the moment it resolves the project (bind_correlation).
job_id	The durable queue job executing (where applicable).	Bound by the JobRunner around each dispatch.
request_id	One HTTP request through the API app (#357).	Bound by the HTTP telemetry middleware for the request's duration: an inbound X-Request-Id is honored when token-shaped (letters/digits/._-, <= 64 chars), else a fresh id is minted; either way it is echoed on the response as X-Request-Id, so a dashboard/client report can be joined to every server-side record of that request. Not part of CORRELATION_KEYS (it is never stamped into job payloads or forwarded headers - run_id does the cross-process work).

The primitives were defined in the Python bloom.logging and are ported in apps/server/src/logger.ts:

• correlation_context(run_id=..., thread_id=..., job_id=...) - scope ids to a block, restoring previous values on exit.
• bind_correlation(...) - bind ids learned mid-flight for the rest of the current task.
• ensure_run_id() / new_run_id() / current_correlation() - mint and read ids.

Propagation across processes

The engine runner is a separate service. RemoteEngine forwards the bound ids on the /implement call as X-Bloom-Run-Id / X-Bloom-Thread-Id headers, and the runner app binds them for the duration of the request - so one run_id filter spans both services' logs.

Event-name taxonomy

Event names are stable, lowercase, dot-separated: <domain>.<subject>[.<outcome>]. Grep by prefix (job., engine.run.) or exact name; never rename an event without updating this table.

Run lifecycle (job runner, bloom.job_runner)

Event	Level	When
run.started	info	A claimed job's dispatch begins (the run's correlation scope opens).
run.completed	info	The dispatch finished and the job was marked done.
run.failed	warning	The dispatch raised; job.retry or job.dead_letter follows.

Durable job queue (bloom.persistence.jobs, bloom.job_runner)

Event	Level	When
job.enqueued	info	A job was accepted (not logged for a deduped enqueue).
job.retry	warning	A failed job was rescheduled with backoff.
job.dead_letter	error	A job exhausted its attempts and was given up on.
job.dead_letter_notify_failed / job.dead_letter_hook_failed	error	The user-facing dead-letter notice itself failed.
job.claim_failed / job.extend_failed	error	Queue claim / lease heartbeat trouble.
job_runner.started	info	The worker pool started.

Engine runs (bloom.engine.runner, bloom.engine.remote, bloom.engine_runner)

Event	Level	When
engine.run.started	info	A policy-governed engine attempt begins (per try).
engine.run.completed	info	The run ended without a failure classification.
engine.run.retried	info	A retryable failure; another attempt follows after backoff.
engine.run.paused	info	A pause-class failure (e.g. usage limit).
engine.run.failed	warning	Terminal failure; escalation is the orchestrator's job.
engine.remote.completed	info	The runner service returned a result over HTTP.
engine.runner.started / engine.runner.completed	info	Inside the runner service process (correlated via the X-Bloom-* headers).

Outbound messaging (bloom.messaging)

Event	Level	When
outbound.message.sent	info	A send was delivered and ledgered (#352).
outbound.message.failed	warning	The transport send failed (recorded, then re-raised).
outbound.ledger_write_failed	error	The audit row failed; delivery is never blocked by it.

HTTP API access (bloom.http)

Event	Level	When
http.request	info; error for 5xx (unhandled exceptions carry the traceback)	One access record per request through the API app, emitted by the telemetry middleware (#357): method, route (the matched route template, unmatched for 404s - never the raw path), status, status_class, duration_ms, plus the ambient correlation ids (request_id always; run_id/thread_id when a handler bound them). Never bodies or headers.

Auth (authRoutes.ts, oidc.ts) - #358

Security events for dashboard sign-in. Bounded metadata only - never a token, code, session id, or password.

Event	Level	When
auth.signin.ok	info	A sign-in completed and a session was minted: method (google for the OIDC callback, test for the non-production test-login - so test-login use is always visible in the trail) and user_id.
auth.signin.fail	warning	A sign-in attempt failed: method plus a bounded reason - provider_error (the IdP redirected back with an error), state_mismatch (missing/expired/forged CSRF state or flow cookie), exchange_failed (the code-for-identity exchange or id_token verification failed). Each is also counted in bloom_auth_signin_failures_total.
auth.oidc.error	warning	The underlying OIDC step that failed inside the provider: stage (token_exchange / token_response / id_token_verify / nonce_mismatch / missing_email) with bounded detail (HTTP status, verifier exception text).
auth.logout	info	Session teardown; carries had_session only - the session id is a bearer secret and never logged.

Credential-store audit (bloom.credentials) - #358

One audit record per touch of the credential surface (M19 write-only store, M20 global store), so "who touched which credential, when, and did it work" is answerable with one credential. filter. Every record carries scope (project/global), actor (the signed-in user's id, or system for deploy-time provisioning), the credential name where one is involved, outcome (ok/error - error logs at warning), the correlation ids, and the pipeline timestamp - never a value (see the redaction guarantee below). Each event is mirrored into bloom_credential_ops_total.

Event	Level	When
credential.write	info; warning on outcome=error	A credential upsert (create/update/rotate) through a dashboard route - per-project or global store.
credential.delete	info; warning on outcome=error	A credential revocation through a dashboard route.
credential.read_metadata	info	A metadata listing (names + provided-at, with count) was served; values have no read path at all (#282).
credential.deliver	info; warning on outcome=error	Deploy-time secret-delivery provisioning in the Infisical client (M19-4): the project's OIDC machine identity was ensured (project, repo, identity_id).

Project events (bloom.events)

Event	Level	When
event.published	info	A project event fanned out to SSE subscribers.

Other modules follow the same <domain>.<subject> shape (e.g. jobs.schema_ready, design_surface.preview_failed); add new names to this table as they are introduced.

Metrics

The metrics layer (#354) lives in apps/server/src/metrics.ts: typed counter/gauge/ histogram helpers backed by prom-client, collected into one process-global registry (getMetrics()), exposed over HTTP by the scrape endpoints below (#355). Call sites depend on BloomMetrics (injectable in tests), never on prometheus_client directly.

Two safety rules keep the registry exposable:

• Correlation ids are never labels. run_id/thread_id/job_id are unbounded; they live on log records (above), and declaring an instrument with one raises. To correlate a metric spike with its runs, pivot on the timestamp and the bounded labels (kind/engine/channel), then filter the logs.
• Bounded cardinality. Each label admits at most 32 distinct values; later values collapse into other (with a metrics.label_cardinality_capped warning), so a misbehaving call site degrades one label instead of growing the registry without bound.

Durations are histograms in seconds, with buckets sized per domain (HTTP calls vs LLM calls vs half-hour engine runs). Counters are exposed with the Prometheus _total suffix.

Metric	Type	Labels	Recorded where / when
bloom_http_requests_total	counter	method, route, status_class	Every request answered by the API app - the telemetry middleware (#357) wraps the whole router stack, so no route is hand-wired. route is the matched route template (/api/projects/{thread_id}, unmatched for 404s), never the raw path; status_class is 2xx..5xx.
bloom_http_request_seconds	histogram	method, route, status_class	Latency of every request through the API app; requests that die on an unhandled exception are observed too (as 5xx).
bloom_http_requests_in_flight	gauge	method	Requests currently being handled (incremented before, decremented after the app runs; route/status are unknowable until routing has run, so method is the only label).
bloom_llm_request_seconds	histogram	provider	Around every LLMProvider.structured call (the factory wraps the built provider in InstrumentedLLMProvider), SDK retries included; failed calls are observed too.
bloom_llm_timeouts_total	counter	provider	A structured call failed on a timeout (detected anywhere in the exception chain, so LLMError-wrapped SDK timeouts count).
bloom_llm_fallbacks_total	counter	primary, fallback	build_llm_provider engaged the configured build-time fallback provider.
bloom_job_queue_depth	gauge	-	Jobs with status pending (incl. those awaiting a retry backoff); sampled by the job runner each claim cycle via JobQueue.pending_count().
bloom_job_retries_total	counter	kind	A failed job was rescheduled with backoff (job.retry).
bloom_job_failures_total	counter	kind	A job dispatch raised (run.failed); each failed attempt counts once, so a dead-lettered job has been counted on every attempt.
bloom_engine_run_seconds	histogram	engine	Duration of one policy-governed EngineRunner.run, its internal retries included (one sample per run).
bloom_engine_runs_total	counter	engine, outcome	Terminal outcome of a run: success (PR opened / changes pushed), escalation (terminal failure needing a human), or the raw non-failure outcome (blocked, skipped).
bloom_ticket_child_parity_total	counter	result	Shadow ticket-child parity checks (#471): at every engine_runs write, the executed child's current_node compared against the #467 projection (result = match/mismatch). Only emitted when BLOOM_TICKET_CHILD_INSTANCES is not disabled.
bloom_channel_sends_total	counter	channel, status	Every outbound send through RecordingMessenger, mirroring the #352 ledger row (status = sent/failed); channel is the transport (telegram today, log in dev, web chat later).
bloom_github_requests_total	counter	method, status	Every GitHub API response, by HTTP method and status code (InstrumentedAsyncClient, used by GitHubClient and App-token minting). Request paths are never labels - they carry per-repo cardinality.
bloom_github_request_seconds	histogram	method	Latency of every GitHub API request, failed attempts included.
bloom_github_request_failures_total	counter	method, reason	timeout/network for transport errors, server_error (5xx) / client_error (4xx) for HTTP-level failures.
bloom_github_rate_limit_remaining	gauge	resource	The x-ratelimit-remaining header of the latest response, per x-ratelimit-resource (usually core) - GitHub degradation must be visible before it bites.
bloom_auth_signin_failures_total	counter	method, reason	Every auth.signin.fail event (#358): method = google/test, reason from the bounded set above - a spike is the operator's brute-force / broken-OIDC signal.
bloom_credential_ops_total	counter	op, scope, outcome	Every credential.* audit event (#358): op = write/delete/read_metadata/deliver, scope = project/global, outcome = ok/error. Credential names are never labels (user-chosen, unbounded); they live on the audit log records.

Never rename a metric or change its label set without updating this table (and the alerting built on these signals - see below).

HTTP request telemetry middleware (#357)

The API's request-level telemetry comes from one HTTP-logging middleware (the bloom.api.telemetry twin: pino-http wiring in apps/server/src/app.ts + apps/server/src/logger.ts), added outermost in the app factory so every router - dashboard APIs, webhooks, health, even /metrics scrapes - is covered without per-route wiring. Per request it emits the http.request access log (taxonomy above), records the three bloom_http_* metrics (table above), and binds the request_id (correlation table above) into the logging context so downstream records on that request correlate. It is hand-rolled on top of the #354 metrics layer rather than starlette-exporter / prometheus-fastapi-instrumentator: those cover only the metrics half (no structured access log or correlation binding) and record outside the guarded BloomMetrics registry.

/metrics scrape endpoints (#355)

Both processes serve their registry in Prometheus text exposition format (bloom.metrics.render_exposition, i.e. prometheus_client's generate_latest plus its Content-Type). Each process has its own registry: the API's carries the orchestrator-side samples (LLM, jobs, channel sends, engine-run outcomes), the runner's the samples its own process records (its GitHub pushes/PR calls).

Endpoint	Where	Exposure
GET /metrics	bloom-server (API container)	Gated: the container binds a publicly reachable edge, so the endpoint is never open in production. With BLOOM_METRICS_TOKEN set, the scraper must send it as a bearer token (authorization: credentials: in the Prometheus scrape config); wrong/missing token is 401. With no token, the endpoint is open in non-production (loopback/dev convenience) and 404s in production - fail closed, indistinguishable from the route not existing.
GET /metrics	engine runner (worker VM)	Bearer-authed with the runner's existing service token (BLOOM_RUNNER_TOKEN), like /health and /implement. The service already binds the VM's tailscale address (never the public edge), so auth is defense-in-depth on top of the tailnet boundary and keeps the runner's single auth model.

The runner already had a /health (bearer-authed, M17), so a scraper can probe liveness and scrape metrics independently of the API. Standing up the Prometheus/Grafana instance that consumes these is out of scope here; the alerting below rides Bloom's own seams instead.

Alerting (#356)

bloom.alerting turns the failure classes the M17 pilot surfaced into proactive operator notifications, instead of leaving them to be discovered by reading chat. Detectors sit on the seams the earlier tickets already instrumented; an AlertManager decides when a recorded failure becomes a notification, and a pluggable AlertSink delivers it.

Failure classes

Class	Severity	Detector seam	Fires when (defaults)
gateway_timeout	warning	InstrumentedLLMProvider (every structured() call, any provider - the OpenClaw gateway included), keyed per provider	3 timeouts within the window
silent_job_failure	critical	JobRunner dead-letter path, keyed per job kind	1 dead-lettered job (it exhausted its retries - work was lost)
runner_unreachable	critical	RunnerHealthMonitor, a background loop probing the engine runner's /health every BLOOM_ALERT_RUNNER_PROBE_INTERVAL_SECONDS (claude_code mode only)	3 consecutive failed probes; a successful probe resets the count
secret_provider_unreachable	critical	InfisicalClient login (network + auth) and secret upsert/delete failures	3 failures within the window - repeated failure here silently breaks deploy-time credential delivery

Every alert carries the correlation ids (run_id/thread_id/job_id, #353) bound where the failure was recorded, and bounded metadata only - never a secret value, credential name, or full free-form body.

Threshold + dedup (the rate-limit lesson)

Each class has a rule: occurrences are counted in a sliding window (BLOOM_ALERT_WINDOW_SECONDS) and an alert fires only when the class's threshold is crossed; after one fires, a cooldown (BLOOM_ALERT_COOLDOWN_SECONDS) suppresses repeats for the same key, and the window restarts - a sustained outage is one notification per cooldown, not a flood. Keys narrow dedup below the class (per LLM provider, per job kind), so distinct sources never suppress each other.

Delivery sinks

Delivery is pluggable (AlertSink); a full external alertmanager integration can slot in behind the same protocol later. The default wiring:

• Operator chat (MessengerAlertSink): when BLOOM_ALERT_CHAT_ID is set, alerts go to that Telegram chat through the same composed messenger as every other outbound send - so each alert is ledgered, logged, and counted like any message (#352). No new channel.
• Structured log (LogAlertSink): with no operator chat configured, alerts land in the log as alert.notified (error for critical, warning otherwise) - dev/test deployments stay externally silent.

Alerting is best-effort by design: a sink failure is logged (alert.delivery_failed) and never raised back into the already-failing code path.

Configuration

Variable	Default	Meaning
BLOOM_ALERTS_ENABLED	true	Master switch for all detectors and the runner monitor.
BLOOM_ALERT_CHAT_ID	unset	Telegram chat id the operator alerts go to; unset routes alerts to the structured log only.
BLOOM_ALERT_WINDOW_SECONDS	600	Sliding window occurrences are counted in.
BLOOM_ALERT_COOLDOWN_SECONDS	900	Minimum gap between fired alerts for one dedup key.
BLOOM_ALERT_GATEWAY_TIMEOUT_THRESHOLD	3	Timeouts (per provider) within the window before an alert.
BLOOM_ALERT_JOB_FAILURE_THRESHOLD	1	Dead-lettered jobs (per kind) within the window before an alert.
BLOOM_ALERT_RUNNER_UNREACHABLE_THRESHOLD	3	Consecutive failed runner health probes before an alert.
BLOOM_ALERT_SECRET_PROVIDER_THRESHOLD	3	Infisical auth/API failures within the window before an alert.
BLOOM_ALERT_RUNNER_PROBE_INTERVAL_SECONDS	60	How often the background monitor probes the runner's /health.

No secrets, no full bodies

Log records never carry credentials or full free-form content:

• Third-party HTTP loggers (httpx, httpcore) are held at WARNING - request URLs can embed bot tokens.
• Free-form content (chat messages, LLM output) is logged as a bounded preview plus a content_sha256 of the full body - bloom.logging.preview() / content_sha256(), the generalized form of the outbound ledger's truncation/hash discipline (#216/#352). The hash makes a record matchable against the stored original without the log stream holding the body.
• Anything derived from an exception on the engine path goes through bloom.engine.command.redact.

Security-surface redaction guarantee (#358)

The auth and credential-store paths carry the highest-value secrets, so their telemetry has a hard guarantee, enforced by tests (tests/integration/test_security_telemetry.py, tests/unit/test_infisical_client.py): no secret value ever appears in a log record or in the metrics exposition on these paths.

• Credential audit events (bloom.credentials) carry name / scope / actor / outcome / correlation ids only - the same write-only discipline as the bloom.infisical client, which logs status codes and names on failure, never a value or response body. Credential names are validated before the audit block, so injection-shaped input never reaches a record, and they are never metric labels.
• Auth events never carry OAuth codes, id_tokens, session ids, or the test-login password; failure records hold a bounded reason (plus a previewed IdP error code / verifier exception text, which never embed request material).
• Metric labels on both surfaces are small closed sets (method/reason, op/scope/outcome) - nothing caller-controlled.