Forge Health

Forge Health — One Signal, Read From Real Calls

How BoB tells “no issues matched” apart from “the lookup failed” (#1748). (Back to reference index.)

Nothing in BoB could previously distinguish an empty answer from an unread one. That is exactly bug #1724: during the 2026-08-17 API wobble, nine of the stale sweep’s reference lookups returned HTTP 503, every one degraded silently to links-unknown, and the sweep then printed “no stale todos — every open todo has at least one open linked issue”. The direction was safe; the claim was not established. An operator reading that clean line concludes the sweep ran, when in fact it mostly did not.

Forge health is the general fix: one classified availability signal, derived from call outcomes that actually happened, that every gh-dependent path can read before it trusts an empty result.

Per the Prime Directive, the signal has exactly one owner (scripts/forge/health.js), it is derived rather than declared, and its verdict is always reported with the evidence behind it.

States

State	Meaning	May an empty result be believed?
operational	Recent calls succeeded.	Yes — empty means empty.
degraded	Some calls failed with server, rate-limit, or auth shapes.	No — empty is indistinguishable from unread.
unreachable	Calls are failing at the network layer, with no successes.	No.
unknown	No evidence inside the window, and probing was suppressed.	No — absence of data is not a verdict.

unknown is deliberately distinct from operational. A signal that guessed “probably fine” when it had never looked would reintroduce the very failure it exists to prevent.

Where the evidence comes from

Classification is derived from real call outcomes — exit codes, error shapes, and latency — never from scraping a status page. A status page reports the provider’s opinion of itself, minutes late, and says nothing about whether this machine, with these credentials can reach the API right now.

Two doors feed the observation store, and both write the same record:

1. The gh PATH shim (bin/gh) — deployed to $BOB_HOME/bin, which already sorts ahead of the real binary on PATH. Every gh invocation anywhere in BoB is classified as a side effect of running, including the prompt-text call sites that run raw gh through Bash. No call site is edited.
2. health.js record — the explicit door, for anything that bypasses the shim, and for tests.

When the store holds no fresh evidence, status issues one active probe (gh api rate_limit — the cheapest real API call there is; it needs no repo context and consumes no quota) and classifies its outcome like any other call.

The signal table

Each call’s exit code and stderr shape map onto one signal. Only the first four move the verdict:

Signal	Recognized from	Availability signal?
ok	exit 0	— (a success)
net	DNS failure, connection refused, timeout, reset	Yes → unreachable
server	HTTP 5xx, Bad gateway, Service Unavailable	Yes → degraded
ratelimit	rate limit exceeded, HTTP 429, abuse detection	Yes → degraded
auth	HTTP 401/403, bad credentials	Yes → degraded
notfound	HTTP 404, “Could not resolve to”	No — a real answer
other	usage errors, unknown flags	No — client-side

A missing issue is not an outage. A 404 is the forge answering correctly, so it must never degrade the verdict — otherwise every legitimate “not found” would poison the signal for everyone.

Nor is every successful call evidence. gh --version succeeds with the network unplugged; recording it as a success would manufacture the exact false confidence this signal exists to destroy. The shim therefore skips the offline-capable subcommands (--version, help, completion, alias, config, extension) and records everything else — so a new gh subcommand is observed by default rather than silently ignored.

An auth failure classifies as degraded rather than as its own state: the forge itself may be perfectly healthy, but this client cannot read it, so an empty result is exactly as untrustworthy. The reason field names the cause.

Classification rules

Applied in order over the most recent 10 observations inside the window:

Condition	Verdict
No observations	unknown
No availability failures	operational
Failures, no successes, network-dominant	unreachable
Failures, no successes	unreachable
Failures alongside successes	degraded

The last row is the #1724 case: some calls worked, which is precisely what makes a silent partial failure so convincing.

The shim

bin/gh is a POSIX-sh wrapper deployed to $BOB_HOME/bin, which already sorts ahead of /opt/homebrew/bin (or /usr/bin) on PATH. It becomes active on the next scripts/deploy.sh — there is nothing to opt into and no call site to edit.

Transparency is the contract. Unknown subcommands, flags, stdin, stdout, stderr and the exit code all pass through unchanged. A health signal that could break gh would be worse than no signal at all, so the shim degrades to a plain exec of the real binary whenever any part of the recording path is unavailable — no state directory, no writable log, no temp file.

Three details worth knowing:

• stderr is only diverted when it is not a terminal. An interactive gh auth login keeps its prompts unbuffered and in order; buffering them to classify the outcome would trade a working login for a data point.
• The real binary is resolved by walking PATH, skipping this file and any other copy of the shim (identified by a marker string in its first bytes). Two copies on PATH cannot find each other and loop; with no real gh reachable at all, the shim exits 127 with a clear message rather than recursing.
• Latency uses a real millisecond clock. GNU date +%s%3N where available, otherwise perl’s Time::HiRes. Whole seconds are the last resort, because a recorded latency of 1000 that actually means “somewhere between 0 and 2s” is the sort of confident-looking number this requirement exists to stamp out.

To bypass the shim for a single call, invoke the real binary by path (/opt/homebrew/bin/gh …); to disable it, remove $BOB_HOME/bin/gh.

The single entry point

One engine serves both worlds, so shell and JS call sites can never drift apart.

Shell — bin/forge-health, whose exit code is the answer:

forge-health                 # 0 operational · 3 degraded · 4 unreachable · 5 unknown
forge-health --json          # the full verdict with its evidence counts
forge-health --no-probe      # read-only: never issues a call
forge-health observations    # the raw evidence behind the verdict

if forge-health --no-probe >/dev/null; then
  : # operational — an empty query result can be trusted
else
  echo "could not check — the forge is not operational"
fi

JS — the same module, require()d:

const forge = require(`${BOB_DIR}/scripts/forge/health.js`);

if (!forge.isTrustworthy()) {
  console.log('could not check — forge unavailable');
}
const { state, reason, sample } = forge.status();

classify() is exported separately and is pure — same observations, same verdict, no I/O — which is what makes the behavior testable without a forge.

The TTL cache

A verdict is cached for 60 seconds (BOB_FORGE_TTL), so N call sites in one cycle cost at most one probe between them. The cache is not allowed to outlive its evidence: recording a new observation invalidates it immediately, so a fresh failure is never masked by a stale “operational”.

Setting	Env	Default
Verdict TTL	BOB_FORGE_TTL	60s
Observation window	BOB_FORGE_WINDOW	300s
Store location	BOB_FORGE_STATE	$BOB_HOME/forge

State lives in $BOB_HOME/forge/: observations.log (a 5-column TSV ring, capped at 500 lines) and health.json (the cached verdict). The TSV format is the shim’s contract — it appends with a single printf, needing no interpreter, no parse, and no lock.

CLI

forge-health status [--json] [--no-probe] [--ttl N] [--window N]
forge-health record --rc N [--ms N] [--signal S] [--sub X]
forge-health observations [--json] [--window N]
forge-health reset

Verification: make test-forge-health (engine) and make test-forge-shim (shim transparency).

The read cache (#1749)

A classified signal keeps a run honest about a degradation; it does not keep the run alive. Warp-drive, /what-next and /groom all block on live issue reads, so a transient 503 kills an autonomous run mid-phase. The read cache closes that: reads fall back to last-known state instead of failing.

It rides the same shim. A successful read is stored on the way past, so the cache is a side effect of normal use — there is no sync step to run or forget, and the ~34 prompt-text gh call sites benefit without being edited.

Store: <project>/.claude/forge/cache/ — per-project, because gh issue list --label req means different things in different repos. It is derived state, not authored state, and is gitignored through the cdprov managed ignore block (see handbook §15.3).

What is cached — an allowlist, not “anything that looks like a read”: issue list, issue view, search issues. Serving a cached answer to issue create would be absurd; serving one to a command nobody has reasoned about is how a cache turns into a liability.

Where the fallback stops matters as much as where it starts:

Situation	Behaviour
Availability failure (net/server/ratelimit/auth), fresh entry	Served from cache, exit 0, marked on stderr
404 / usage error	Not served — the forge answered correctly, and a stale copy of a deleted issue is a wrong answer, not a resilient one
Query never seen live	Fails honestly — it must never invent an empty result
Entry past the staleness bound	Reported as a condition; the failure is passed through rather than a day-old list handed over as current
Any mutation	Never cached, never served

A cached read is never presented as live. The payload goes to stdout untouched — a caller piping into jq is unaffected — and the marker goes to stderr:

[forge-cache] served from cache (age 45s) — the live call failed (server). This is last-known state, not live.

The event is also recorded, so forge-health reports a cache block that /what-next and the fleet snapshot surface. That condition is reported separately from availability, and checked even when the state is clean: the forge can be operational right now while the answers a run acted on came from cache minutes ago.

Setting	Env	Default
Staleness bound	BOB_FORGE_CACHE_MAX_AGE	86400 (24h)
Disable entirely	BOB_FORGE_CACHE=0	enabled

node scripts/forge/cache.js status    # what is cached, and how old
node scripts/forge/cache.js clear     # drop it; costs a re-fetch, nothing more

Verification: make test-forge-cache (the store) and make test-forge-cache-shim (the behaviour through a raw gh call).

The write queue (#1750)

Reads can fall back to cache. Writes cannot be invented — and warp-drive writes constantly: checking off acceptance criteria, filing todos and journals, commenting, closing. Every one of those is a gh issue mutation, and during a degradation every one of them simply fails. A degraded run does not merely stall; it loses PM state that no later read can reconstruct.

The write-ahead queue is the spool that stops that. A mutation that cannot be delivered is written down durably, in order, with everything needed to deliver it later.

Store: <project>/.claude/forge/queue/ — beside #1749’s cache, under the same one ignored .claude/forge/ path in the cdprov managed ignore block (#1479). It is spool state, not authored state: nothing in it is written by a human, nothing is read as a source of truth, and it never competes with GitHub for truth. It must outlive the session, which is exactly why it does not live inside .claude/.warp-drive-state.json — a queue that died with a degraded run would lose every pending mutation at the worst possible moment.

What is spooled — an allowlist, matching what warp-drive actually writes: issue create, issue comment, issue edit (which carries label add/remove), issue close, issue reopen. issue delete, issue transfer and issue lock are deliberately absent: replaying a destructive or cross-repo move hours later, against a forge whose state has moved on, is damage rather than resilience. They fail honestly during an outage.

The payload is captured, not just the command line. Warp-drive writes almost everything through --body-file -, whose stdin is gone the moment the process exits — an argv-only spool would replay into an empty body. Body text is therefore materialized into the entry’s own sidecar at enqueue time (from --body, --body-file, or stdin alike) and the argv rewritten to point at it.

Bounded by decision. The spool holds at most BOB_FORGE_QUEUE_MAX pending entries (default 500). At the bound, enqueue refuses and the caller sees the real failure. Unbounded growth turns a long outage into a disk problem; silently dropping the oldest write would be the exact failure this whole area exists to eliminate. Failing loudly at a known edge is the honest third option.

Layout — one directory per project, one file per mutation:

File	Holds
NNNNNN.seq	The allocation marker. Created O_EXCL — that one atomic syscall is the entire concurrency story, so two gh processes racing during an outage each get a distinct slot in a stable order.
NNNNNN.json	The entry: kind, repo, target, handle, idempotency key, replayable argv. Written temp-then-rename, so a reader never sees a torn record.
NNNNNN.d/body.md	The materialized body, verbatim — byte-identical to what the live call sent.
handles.json	BOB-Q<seq> → the real issue number, once its create lands.
replay.log	TSV of every replay attempt and its outcome.

Spooled by the shim, not by a library

Warp-drive’s issue writes are almost entirely prompt-level: gh issue comment, gh issue edit, gh issue create written into command and skill markdown and run through Bash. There is no library boundary to hook. So the same bin/gh shim that classifies availability and serves the read cache is what enqueues them — no call site is edited, exactly as in #1748.

The shim does three things before the real binary runs, because two of them change what the real binary is asked to do:

1. Resolve handles. Any argument that is exactly BOB-Q<n> is looked up. If its create has already replayed, the real number is substituted in place and the call goes live — no call site aware anything happened.
2. Capture stdin. If the body arrives on --body-file -, it is slurped to a temp file first and the argv rewritten to point at it. The pipe is gone the moment the process exits; the spool has to own the text before the call is even attempted.
3. Decide spoolability. A bare gh issue create (no --title) opens an interactive editor and is left strictly alone — there is no payload to spool, and buffering its stderr to classify the outcome would trade a working prompt for a data point.

Then, if the call fails:

Situation	Behaviour
Availability failure (net/server/ratelimit/auth)	Spooled; exit 0, marked on stderr
404 / usage error	Not spooled — the forge answered correctly, and replaying a write it already refused is not resilience
Target is an unresolved BOB-Q<n>	Spooled without calling the forge at all, behind the create it depends on — even when the forge is healthy
Spool at its bound	Not queued, not delivered: the caller gets the real failure, loudly attributed
Any read	Never spooled — #1749 already answers those

A spooled write exits 0 but is never presented as delivered. Exiting non-zero would kill the run this spool exists to keep alive; claiming delivery would be the silent loss it exists to prevent. So the marker goes to stderr, the same contract the read cache uses:

[forge-queue] not delivered yet — the live call failed (server). Spooled as q000003; it replays in order when the forge recovers (queue depth 3).

A queued create additionally prints its handle to stdout, where the real issue URL would have gone:

https://github.com/acme/widget/issues/BOB-Q1

That is load-bearing rather than cosmetic. Warp-drive files a todo and references its number in the next breath; basename of that URL yields BOB-Q1, which is accepted anywhere an issue number is accepted for the rest of the run and resolves to the real number once the create replays. A spool that accepted the create but broke the reference would trade one silent loss for another.

Setting	Env	Default
Pending-entry bound	BOB_FORGE_QUEUE_MAX	500
Disable entirely	BOB_FORGE_QUEUE=0	enabled

node scripts/forge/queue.js status    # depth, oldest entry, unresolved handles
node scripts/forge/queue.js list      # what is pending, in replay order
node scripts/forge/queue.js show 3    # one entry, body included

Replay

The queue drains on recovery, triggered by the same thing that populates the read cache: normal use. The first gh call to succeed after an outage notices the spool and drains it, so there is no daemon, no cron, and no step for a human to forget. The gate is a single stat — the engine leaves a breadcrumb in the global forge state dir naming the project roots with something pending, so the overwhelming majority of calls, which have nothing to drain, pay nothing.

Replay runs the real binary directly, bypassing the shim. That is not tidiness: a replay that went back through the shim could be re-spooled on failure, and the queue would grow every time it tried to drain itself. The attempt is still recorded as an observation, through health.js record — the documented door for anything that goes around the shim.

Idempotency is by content, not by hope. A mutation that reached GitHub microseconds before the connection dropped is, from the caller’s side, indistinguishable from one that never arrived. So before delivering, replay looks:

Kind	Check
comment	The issue’s comments, for one whose body already matches the spooled body exactly
create	Recent issues (read from the list endpoint, not search — search indexing lags, and an index that has not caught up would report “not created” for an issue that exists) for a matching body, or title when the create carried no body
close / reopen	The issue’s current state — already there means already applied
edit	None needed: setting a label, title or body is a set operation, so replaying it lands the same final state

An earlier design stamped a <!-- bob-idem:... --> marker into every spooled body so replay could recognise its own writes. It was dropped, because the marker can only ever appear on a body the replay sent — never on one that partially landed from the live call, which is the case that matters most. Stamping the live call instead would mean mutating every issue body BoB writes, healthy or degraded, to serve a failure path. Exact content matching covers both cases and changes nothing about what a healthy run produces.

Two kinds of stop, deliberately different:

• The forge is still down (an availability failure, at delivery or at the verification read): the whole replay stops. The remaining entries lose nothing by waiting, and pushing the rest of the queue at a degraded API would make the outage worse.
• One entry cannot be delivered on its own terms — its target was deleted during the outage, it exhausted its attempts, or the create it depends on never landed: only that entry is set aside, with a reason, and the queue continues. That is AC-07: per item, not per queue. A set-aside entry stays on disk and is reported by status; it is never discarded.

Replay is dry-run by default, matching bob-reap and bob-mirror. Its exit code is the answer: 0 only when the spool drained cleanly and completely — a skip, a failure, or a mid-queue stop all exit 3, because reporting any of them as success is the silent-loss failure mode in miniature.

forge-queue replay            # what it would deliver
forge-queue replay --apply    # deliver it

Setting	Env	Default
Attempts before an entry is set aside	BOB_FORGE_QUEUE_ATTEMPTS	3
Entries drained per automatic replay	BOB_FORGE_QUEUE_BATCH	25 (--batch 0 = all)
Disable automatic replay	BOB_FORGE_REPLAY=0	enabled

Seeing it

A queued write is PM state that has not landed. It is therefore surfaced wherever a human already looks, and never left to be discovered by reading a directory:

Surface	What it shows
forge-queue	Depth, the oldest entry’s age, the pending mix by kind, unresolved handles, and the last replay outcome. Exit code answers “is anything undelivered?”
warp status	A Forge queue: line — only when there is something to say, so a clean run does not grow a permanent “queue: 0”. Yellow while entries are merely waiting; red the moment one is skipped or failed.
Fleet snapshot (audit.js snapshot / view)	summary.forge_queue per machine and forge_queue per project, rendered directly beneath the availability table — it answers the other half of the same question: not “could we read” but “did everything we wrote actually land”.

The distinction the surfaces keep is the one that matters: pending entries replay on the next successful gh call and need nobody; skipped and failed ones will not arrive on their own and need a human. They are counted separately everywhere, and a failed replay is never reduced to a number — the reason travels with it.

Verification: make test-forge-queue (the store and the observable shape), make test-forge-queue-shim (the behaviour through a raw gh call), and make test-forge-queue-replay — which drives a stateful mock forge, because without state there is no way to tell “replayed correctly” from “replayed twice”.

Consumers

Surface	What it does with the signal
Stale todo sweep (todo-consume.js stale)	Prints UNCHECKED #NN for todos whose refs could not be read; the summary reports how many could not be fully checked instead of asserting every link is open.
/what-next	Step 0 checks the condition before any gh read, leads the report with it when impaired, and never reports “no work found” while the forge is not operational. Also gates the health verdict — idle and “could not read the queue” are the same empty list.
Fleet snapshot (audit.js snapshot / view)	Carries summary.forge per machine and renders a Forge section, including cached reads next to an otherwise-clean verdict. Deliberately per-machine: network path and credentials differ per host, so “the forge is up” is not a fleet-wide fact.
The read cache (bin/gh → scripts/forge/cache.js)	Falls back to last-known state on an availability failure, so a run survives a degradation instead of dying mid-phase.
The write queue (bin/gh → scripts/forge/queue.js)	Spools a mutation it cannot deliver and replays it in order on recovery, so a degraded run loses no PM state. Uses the same net/server/ratelimit/auth boundary the cache does.

New consumers should ask isTrustworthy() rather than re-deriving the rule. The question is always the same one — may this empty result be believed?

Related

• Mirror Remote — the same resilience family, at the git layer.
• TODO Format Contract — the stale sweep, forge health’s first consumer: it now prints UNCHECKED #NN for todos whose refs could not be read, and its closing line reports what it could not establish rather than asserting every link is open.