Forge Health

Forge Health — One Signal, Read From Real Calls

How BoB tells “no issues matched” apart from “the lookup failed” (#1748). (Back to reference index.)

Nothing in BoB could previously distinguish an empty answer from an unread one. That is exactly bug #1724: during the 2026-08-17 API wobble, nine of the stale sweep’s reference lookups returned HTTP 503, every one degraded silently to links-unknown, and the sweep then printed “no stale todos — every open todo has at least one open linked issue”. The direction was safe; the claim was not established. An operator reading that clean line concludes the sweep ran, when in fact it mostly did not.

Forge health is the general fix: one classified availability signal, derived from call outcomes that actually happened, that every gh-dependent path can read before it trusts an empty result.

Per the Prime Directive, the signal has exactly one owner (scripts/forge/health.js), it is derived rather than declared, and its verdict is always reported with the evidence behind it.

States

State	Meaning	May an empty result be believed?
operational	Recent calls succeeded.	Yes — empty means empty.
degraded	Some calls failed with server, rate-limit, or auth shapes.	No — empty is indistinguishable from unread.
unreachable	Calls are failing at the network layer, with no successes.	No.
unknown	No evidence inside the window, and probing was suppressed.	No — absence of data is not a verdict.

unknown is deliberately distinct from operational. A signal that guessed “probably fine” when it had never looked would reintroduce the very failure it exists to prevent.

Where the evidence comes from

Classification is derived from real call outcomes — exit codes, error shapes, and latency — never from scraping a status page. A status page reports the provider’s opinion of itself, minutes late, and says nothing about whether this machine, with these credentials can reach the API right now.

Two doors feed the observation store, and both write the same record:

1. The gh PATH shim (bin/gh) — deployed to $BOB_HOME/bin, which already sorts ahead of the real binary on PATH. Every gh invocation anywhere in BoB is classified as a side effect of running, including the prompt-text call sites that run raw gh through Bash. No call site is edited.
2. health.js record — the explicit door, for anything that bypasses the shim, and for tests.

When the store holds no fresh evidence, status issues one active probe (gh api rate_limit — the cheapest real API call there is; it needs no repo context and consumes no quota) and classifies its outcome like any other call.

The signal table

Each call’s exit code and stderr shape map onto one signal. Only the first four move the verdict:

Signal	Recognized from	Availability signal?
ok	exit 0	— (a success)
net	DNS failure, connection refused, timeout, reset	Yes → unreachable
server	HTTP 5xx, Bad gateway, Service Unavailable	Yes → degraded
ratelimit	rate limit exceeded, HTTP 429, abuse detection	Yes → degraded
auth	HTTP 401/403, bad credentials	Yes → degraded
notfound	HTTP 404, “Could not resolve to”	No — a real answer
other	usage errors, unknown flags	No — client-side

A missing issue is not an outage. A 404 is the forge answering correctly, so it must never degrade the verdict — otherwise every legitimate “not found” would poison the signal for everyone.

Nor is every successful call evidence. gh --version succeeds with the network unplugged; recording it as a success would manufacture the exact false confidence this signal exists to destroy. The shim therefore skips the offline-capable subcommands (--version, help, completion, alias, config, extension) and records everything else — so a new gh subcommand is observed by default rather than silently ignored.

An auth failure classifies as degraded rather than as its own state: the forge itself may be perfectly healthy, but this client cannot read it, so an empty result is exactly as untrustworthy. The reason field names the cause.

Classification rules

Applied in order over the most recent 10 observations inside the window:

Condition	Verdict
No observations	unknown
No availability failures	operational
Failures, every one a rate limit (with or without successes)	degraded (rate-limited) — the forge is up and the quota ran out, so it is never unreachable (#3191)
Failures, no successes, network-dominant	unreachable
Failures, no successes	unreachable
Failures alongside successes	degraded

The last row is the #1724 case: some calls worked, which is precisely what makes a silent partial failure so convincing.

The shim

bin/gh is a POSIX-sh wrapper deployed to $BOB_HOME/bin, which already sorts ahead of /opt/homebrew/bin (or /usr/bin) on PATH. It becomes active on the next scripts/deploy.sh — there is nothing to opt into and no call site to edit.

Transparency is the contract. Unknown subcommands, flags, stdin, stdout, stderr and the exit code all pass through unchanged. A health signal that could break gh would be worse than no signal at all, so the shim degrades to a plain exec of the real binary whenever any part of the recording path is unavailable — no state directory, no writable log, no temp file.

Three details worth knowing:

• stderr is only diverted when it is not a terminal. An interactive gh auth login keeps its prompts unbuffered and in order; buffering them to classify the outcome would trade a working login for a data point.
• The real binary is resolved by walking PATH, skipping this file and any other copy of the shim (identified by a marker string in its first bytes). Two copies on PATH cannot find each other and loop; with no real gh reachable at all, the shim exits 127 with a clear message rather than recursing.
• Latency uses a real millisecond clock. GNU date +%s%3N where available, otherwise perl’s Time::HiRes. Whole seconds are the last resort, because a recorded latency of 1000 that actually means “somewhere between 0 and 2s” is the sort of confident-looking number this requirement exists to stamp out.

To bypass the shim for a single call, invoke the real binary by path (/opt/homebrew/bin/gh …); to disable it, remove $BOB_HOME/bin/gh.

The single entry point

One engine serves both worlds, so shell and JS call sites can never drift apart.

Shell — bob forge health (engine scripts/cli/forge-health; bin/forge-health is its alias shim, #2060), whose exit code is the answer:

bob forge health                 # 0 operational · 3 degraded · 4 unreachable · 5 unknown
bob forge health --json          # the full verdict with its evidence counts
bob forge health --no-probe      # read-only: never issues a call
bob forge health observations    # the raw evidence behind the verdict

if forge-health --no-probe >/dev/null; then
  : # operational — an empty query result can be trusted
else
  echo "could not check — the forge is not operational"
fi

JS — the same module, require()d:

const forge = require(`${BOB_DIR}/scripts/forge/health.js`);

if (!forge.isTrustworthy()) {
  console.log('could not check — forge unavailable');
}
const { state, reason, sample } = forge.status();

classify() is exported separately and is pure — same observations, same verdict, no I/O — which is what makes the behavior testable without a forge.

The TTL cache

A verdict is cached for 60 seconds (BOB_FORGE_TTL), so N call sites in one cycle cost at most one probe between them. The cache is not allowed to outlive its evidence: recording a new observation invalidates it immediately, so a fresh failure is never masked by a stale “operational”.

Setting	Env	Default
Verdict TTL	BOB_FORGE_TTL	60s
Observation window	BOB_FORGE_WINDOW	300s
Store location	BOB_FORGE_STATE	$BOB_HOME/forge

State lives in $BOB_HOME/forge/: observations.log (a TSV ring, capped at 5,000 lines) and health.json (the cached verdict). The TSV format is the shim’s contract — it appends with a single printf, needing no interpreter, no parse, and no lock.

An observation has nine columns: epoch, exit code, latency in ms, signal, subcommand, then (#3190) the quota the call spends, its cost, the caller and the session. A line an older shim wrote has the first five and still parses. The quota files beside it are described under The quota budget.

CLI

bob forge health status [--json] [--no-probe] [--ttl N] [--window N]
bob forge health record --rc N [--ms N] [--signal S] [--sub X]
bob forge health observations [--json] [--window N]
bob forge health reset

status also prints the GitHub quota (#3190).

Verification: make test-forge-health (engine) and make test-forge-shim (shim transparency).

The read cache (#1749)

A classified signal keeps a run honest about a degradation; it does not keep the run alive. Warp-drive, /what-next and /groom all block on live issue reads, so a transient 503 kills an autonomous run mid-phase. The read cache closes that: reads fall back to last-known state instead of failing.

It rides the same shim. A successful read is stored on the way past, so the cache is a side effect of normal use — there is no sync step to run or forget, and the ~34 prompt-text gh call sites benefit without being edited.

Store: <project>/.claude/forge/cache/ — per-project, because gh issue list --label req means different things in different repos. It is derived state, not authored state, and is gitignored through the bob provision managed ignore block (see handbook §15.3). It holds issue text, which for a private repo is private: the folder is mode 700 and every file 600, whatever the umask. A folder an older version left open is closed, with the files already in it, on the next write (#3636).

What is cached — an allowlist, not “anything that looks like a read”: issue list, issue view, search issues, and label list (#3195 — the bob ready label check reads last-known labels under a rate limit). Serving a cached answer to issue create would be absurd; serving one to a command nobody has reasoned about is how a cache turns into a liability.

Where the fallback stops matters as much as where it starts:

Situation	Behaviour
Availability failure (net/server/ratelimit/auth), fresh entry	Served from cache, exit 0, marked on stderr
404 / usage error	Not served — the forge answered correctly, and a stale copy of a deleted issue is a wrong answer, not a resilient one
Query never seen live	Fails honestly — it must never invent an empty result
Entry past the staleness bound	Reported as a condition; the failure is passed through rather than a day-old list handed over as current
Any mutation	Never cached, never served

A cached read is never presented as live. The payload goes to stdout untouched — a caller piping into jq is unaffected — and the marker goes to stderr:

[forge-cache] served from cache (age 45s) — the live call failed (server). This is last-known state, not live.

The event is also recorded, so bob forge health reports a cache block that /what-next and the fleet snapshot surface. That condition is reported separately from availability, and checked even when the state is clean: the forge can be operational right now while the answers a run acted on came from cache minutes ago.

Setting	Env	Default
Staleness bound	BOB_FORGE_CACHE_MAX_AGE	86400 (24h)
Disable entirely	BOB_FORGE_CACHE=0	enabled

node scripts/forge/cache.js status    # what is cached, and how old
node scripts/forge/cache.js clear     # drop it; costs a re-fetch, nothing more

Verification: make test-forge-cache (the store) and make test-forge-cache-shim (the behaviour through a raw gh call).

The write queue (#1750)

Reads can fall back to cache. Writes cannot be invented — and warp-drive writes constantly: checking off acceptance criteria, filing todos and journals, commenting, closing. Every one of those is a gh issue mutation, and during a degradation every one of them simply fails. A degraded run does not merely stall; it loses PM state that no later read can reconstruct.

The write-ahead queue is the spool that stops that. A mutation that cannot be delivered is written down durably, in order, with everything needed to deliver it later.

Store: <project>/.claude/forge/queue/ — beside #1749’s cache, private in the same way (folders 700, files 600, #3636), and under the same one ignored .claude/forge/ path in the bob provision managed ignore block (#1479). It is spool state, not authored state: nothing in it is written by a human, nothing is read as a source of truth, and it never competes with GitHub for truth. It must outlive the session, which is exactly why it does not live inside .claude/.warp-drive-state.json — a queue that died with a degraded run would lose every pending mutation at the worst possible moment.

What is spooled — an allowlist, matching what warp-drive actually writes: issue create, issue comment, issue edit (which carries label add/remove), issue close, issue reopen. issue delete, issue transfer and issue lock are deliberately absent: replaying a destructive or cross-repo move hours later, against a forge whose state has moved on, is damage rather than resilience. They fail honestly during an outage.

The payload is captured, not just the command line. Warp-drive writes almost everything through --body-file -, whose stdin is gone the moment the process exits — an argv-only spool would replay into an empty body. Body text is therefore materialized into the entry’s own sidecar at enqueue time (from --body, --body-file, or stdin alike) and the argv rewritten to point at it.

An empty edit is never carried (#3186). issue edit replaces the issue body, so an edit with an empty or whitespace-only body would wipe it on replay, hours later, with nobody watching. An empty body is almost always a failed read feeding a write: gh issue view N -q .body > b.md fails in the same outage and leaves b.md empty, then gh issue edit N --body-file b.md follows. The queue refuses that edit: the command fails with the forge’s own error and a [forge-queue] NOT queued and NOT delivered line that says why. A body file that cannot be read is refused for every kind of write. Replay applies the same rule to an edit already in a spool and sets it aside as skipped.

Only the checkout’s own repo (#3635). The queue carries a write only when it is for the repo this checkout’s origin remote names. A write with a --repo that names another repo is refused, and so is any write from a checkout with no origin: the command fails with the forge’s own error and a [forge-queue] NOT queued and NOT delivered line. Replay refuses an entry for another repo, so the queue does not accept one it would later refuse.

Bounded by decision. The spool holds at most BOB_FORGE_QUEUE_MAX pending entries (default 500). At the bound, enqueue refuses and the caller sees the real failure. Unbounded growth turns a long outage into a disk problem; silently dropping the oldest write would be the exact failure this whole area exists to eliminate. Failing loudly at a known edge is the honest third option.

Layout — one directory per project, one file per mutation:

File	Holds
NNNNNN.seq	The allocation marker. Created O_EXCL — that one atomic syscall is the entire concurrency story, so two gh processes racing during an outage each get a distinct slot in a stable order.
NNNNNN.json	The entry: kind, repo, target, handle, idempotency key, replayable argv. Written temp-then-rename, so a reader never sees a torn record.
NNNNNN.d/body.md	The materialized body, verbatim — byte-identical to what the live call sent.
handles.json	BOB-Q<seq> → the real issue number, once its create lands.
replay.log	TSV of every replay attempt and its outcome.

Spooled by the shim, not by a library

Warp-drive’s issue writes are almost entirely prompt-level: gh issue comment, gh issue edit, gh issue create written into command and skill markdown and run through Bash. There is no library boundary to hook. So the same bin/gh shim that classifies availability and serves the read cache is what enqueues them — no call site is edited, exactly as in #1748.

The shim does three things before the real binary runs, because two of them change what the real binary is asked to do:

1. Resolve handles. Any argument that is exactly BOB-Q<n> is looked up. If its create has already replayed, the real number is substituted in place and the call goes live — no call site aware anything happened.
2. Capture stdin. If the body arrives on --body-file -, it is slurped to a temp file first and the argv rewritten to point at it. The pipe is gone the moment the process exits; the spool has to own the text before the call is even attempted.
3. Decide spoolability. A bare gh issue create (no --title) opens an interactive editor and is left strictly alone — there is no payload to spool, and buffering its stderr to classify the outcome would trade a working prompt for a data point.

Then, if the call fails:

Situation	Behaviour
Availability failure (net/server/ratelimit/auth)	Spooled; exit 0, marked on stderr
404 / usage error	Not spooled — the forge answered correctly, and replaying a write it already refused is not resilience
Target is an unresolved BOB-Q<n>	Spooled without calling the forge at all, behind the create it depends on — even when the forge is healthy
Spool at its bound	Not queued, not delivered: the caller gets the real failure, loudly attributed
The write names a repo other than this checkout’s origin (#3635)	Not queued, not delivered: the caller gets the real failure and the reason
Any read	Never spooled — #1749 already answers those

A spooled write exits 0 but is never presented as delivered. Exiting non-zero would kill the run this spool exists to keep alive; claiming delivery would be the silent loss it exists to prevent. So the marker goes to stderr, the same contract the read cache uses:

[forge-queue] not delivered yet — the live call failed (server). Spooled as q000003; it replays in order when the forge recovers (queue depth 3).

A queued create additionally prints its handle to stdout, where the real issue URL would have gone:

https://github.com/acme/widget/issues/BOB-Q1

That is load-bearing rather than cosmetic. Warp-drive files a todo and references its number in the next breath; basename of that URL yields BOB-Q1, which is accepted anywhere an issue number is accepted for the rest of the run and resolves to the real number once the create replays. A spool that accepted the create but broke the reference would trade one silent loss for another.

Setting	Env	Default
Pending-entry bound	BOB_FORGE_QUEUE_MAX	500
Disable entirely	BOB_FORGE_QUEUE=0	enabled

node scripts/forge/queue.js status    # depth, oldest entry, unresolved handles
node scripts/forge/queue.js list      # what is pending, in replay order
node scripts/forge/queue.js show 3    # one entry, body included
node scripts/forge/queue.js drop 3    # set one pending entry aside; the rest stay queued

drop <id> takes 3, #3 or q000003. It marks the entry dropped and keeps its sequence number, so a handle is never reused and list --all still shows it. clear removes every entry; use drop when one entry is wrong.

Replay

The queue drains on recovery, triggered by the same thing that populates the read cache: normal use. The first gh call to succeed after an outage notices the spool and drains it, so there is no daemon, no cron, and no step for a human to forget. The gate is a single stat — the engine leaves a breadcrumb in the global forge state dir naming the project roots with something pending, so the overwhelming majority of calls, which have nothing to drain, pay nothing.

Automatic replay needs this project’s mark (#3635). The breadcrumb is the file queue-dirty in the forge state dir ($BOB_FORGE_STATE, default $BOB_HOME/forge). Enqueue adds the project’s real path to it. The shim’s automatic replay and the warp-drive item-boundary replay deliver only when this project’s real path is one of its lines (replay --marked-only). A queue directory that arrived with a checkout has no mark, so nothing replays it on its own. bob forge queue status prints a note: line when entries are pending and the project is not marked. A replay run by hand (bob forge queue replay --apply) does not need the mark; it still runs every check below.

What replay checks before it delivers (#3635)

Replay runs gh with your token, hours after the entry was written. An entry is a file in the working tree, so replay checks each one again before any call is made for it. An entry is refused when any of these holds:

Check	Refused when
Git	Git tracks the entry (NNNNNN.json, NNNNNN.seq or a file in NNNNNN.d/), or tracks handles.json. The queue directory is ignored by git, so a tracked entry came from a commit, not from a failed write on this machine. A tracked handles.json refuses every entry.
Command	The argv is not one of the queueable shapes (issue create, comment, edit, close, reopen), or is not the kind and the target the entry records.
Repo	The entry’s recorded repo is not this checkout’s current origin, the checkout has no origin, or any --repo in the argv names a different repo.
Body file	A --body-file in the argv, or the entry’s body_file, is outside the entry’s own NNNNNN.d/ folder once symlinks are followed.

A refused entry is never sent. Replay prints it with the reason, sets it aside with status refused, and carries on with the next entry:

  #1 refused         its command is not a queueable issue mutation (issue create, comment, edit, close or reopen)
  #2 delivered       gh comment delivered

Replayed: 1 delivered, 0 skipped, 0 failed, 1 refused, 0 still pending.
1 entry was refused and NOT sent — inspect with: forge-queue show <seq>

A dry run prints the same refused rows and changes nothing. A refused entry stays on disk: bob forge queue list --all lists it and show <seq> prints it with the reason. If the write was a real one (for example the queue directory was committed by mistake), run the command again by hand.

Replay runs the real binary directly, bypassing the shim. That is not tidiness: a replay that went back through the shim could be re-spooled on failure, and the queue would grow every time it tried to drain itself. The attempt is still recorded as an observation, through health.js record — the documented door for anything that goes around the shim.

Idempotency is by content, not by hope. A mutation that reached GitHub microseconds before the connection dropped is, from the caller’s side, indistinguishable from one that never arrived. So before delivering, replay looks:

Kind	Check
comment	The issue’s comments, for one whose body already matches the spooled body exactly
create	Recent issues (read from the list endpoint, not search — search indexing lags, and an index that has not caught up would report “not created” for an issue that exists) for a matching body, or title when the create carried no body
close / reopen	The issue’s current state — already there means already applied
edit	None needed: setting a label, title or body is a set operation, so replaying it lands the same final state

An earlier design stamped a <!-- bob-idem:... --> marker into every spooled body so replay could recognise its own writes. It was dropped, because the marker can only ever appear on a body the replay sent — never on one that partially landed from the live call, which is the case that matters most. Stamping the live call instead would mean mutating every issue body BoB writes, healthy or degraded, to serve a failure path. Exact content matching covers both cases and changes nothing about what a healthy run produces.

Two kinds of stop, deliberately different:

• The forge is still down (an availability failure, at delivery or at the verification read): the whole replay stops. The remaining entries lose nothing by waiting, and pushing the rest of the queue at a degraded API would make the outage worse.
• One entry cannot be delivered on its own terms — its target was deleted during the outage, it exhausted its attempts, or the create it depends on never landed: only that entry is set aside, with a reason, and the queue continues. That is AC-07: per item, not per queue. A set-aside entry stays on disk and is reported by status; it is never discarded.

A refused entry (above) is set aside the same way, before any call is made for it.

Replay is dry-run by default, matching bob reap and bob mirror. Its exit code is the answer: 0 only when the spool drained cleanly and completely — a skip, a failure, a refusal or a mid-queue stop all exit 3, because reporting any of them as success is the silent-loss failure mode in miniature.

bob forge queue replay            # what it would deliver
bob forge queue replay --apply    # deliver it

Setting	Env	Default
Attempts before an entry is set aside	BOB_FORGE_QUEUE_ATTEMPTS	3
Entries drained per automatic replay	BOB_FORGE_QUEUE_BATCH	25 (--batch 0 = all)
Disable automatic replay	BOB_FORGE_REPLAY=0	enabled

Seeing it

A queued write is PM state that has not landed. It is therefore surfaced wherever a human already looks, and never left to be discovered by reading a directory:

Surface	What it shows
bob forge queue	Depth, the oldest entry’s age, the pending mix by kind, unresolved handles, and the last replay outcome. Exit code answers “is anything undelivered?”
bob warp status	A Forge queue: line — only when there is something to say, so a clean run does not grow a permanent “queue: 0”. Yellow while entries are merely waiting; red the moment one is skipped or failed.
Fleet snapshot (audit.js snapshot / view)	summary.forge_queue per machine and forge_queue per project, rendered directly beneath the availability table — it answers the other half of the same question: not “could we read” but “did everything we wrote actually land”.

The distinction the surfaces keep is the one that matters: pending entries replay on the next successful gh call and need nobody; skipped, failed and refused ones will not arrive on their own and need a human. They are counted separately everywhere, and a failed replay is never reduced to a number — the reason travels with it.

Verification: make test-forge-queue (the store and the observable shape), make test-forge-queue-shim (the behaviour through a raw gh call), and make test-forge-queue-replay — which drives a stateful mock forge, because without state there is no way to tell “replayed correctly” from “replayed twice”.

The quota budget (#3190)

The cache and the queue let a run survive a rate limit. The budget exists so that no BoB caller meets one. Every session on a machine — warp runs, headless bob groom, briefs, fleet snapshots — spends the same GitHub token’s quota: 5,000 GraphQL points and 5,000 REST requests an hour, plus burst limits on how fast requests may arrive. The shim keeps the remaining quota in one file and gates every call on it before the call reaches the network.

One shared file

$BOB_HOME/forge/budget.env (bob-forge-budget/1) holds, per quota (graphql, core, search), the limit, what remains, what was used and when it resets, plus fetched_at. It is plain key=value lines, so the shim reads it without starting an interpreter.

The shim fills it from gh api rate_limit, an endpoint that costs no quota:

• at most once every BOB_FORGE_BUDGET_TTL seconds (default 60), for every session together — a lock directory (budget.lock) is taken atomically, and a caller that finds it held uses the number already on file;
• every 15 seconds once a quota is within twice its floor;
• again after a wait, before the call is released.

A lock older than 30 seconds belonged to a caller that died and is taken over. A quota that cannot be read fails open: the call runs. It is never held back on a number nobody has.

Which quota a call spends is decided in the shim: gh api graphql and gh’s own commands (issue, pr, repo, project, …) spend graphql; gh api <path> and run, workflow, release, secret, variable spend core; gh search spends search. gh api rate_limit, gh auth and the offline commands spend nothing and are never gated.

Pacing

Remaining quota	Non-interactive caller	Interactive caller
At or above the pace line	runs at once	runs at once
Under the pace line	one short sleep (BOB_FORGE_PACE_SECONDS), then runs	runs at once
Under the floor	sleeps until the reset, re-reads the quota, then runs	warned on stderr, runs at once

A caller is interactive only when both stdin and stderr are a terminal. An agent’s Bash call, a farm tick and bob groom are not, so they wait; a person who typed the command gets their answer and a line saying the quota is low.

The wait is bounded by BOB_FORGE_MAX_WAIT in total. A reset further away than that is not waited for: the call goes ahead, and a limit, if it comes, is handled by the backoff below.

While it waits the shim prints one line on stderr:

[forge-budget] waiting 412s for the GitHub graphql quota to reset at 10:47 (120 left, floor 300). This call finishes on its own: do not retry it and do not stop.

The line tells a sub-agent what to do, so it never has to judge whether to wait.

Per-minute pacing (#3978)

The hourly quota is not GitHub’s only limit. Its secondary limit counts how fast calls arrive. On 2026-10-04 one machine was held for 40 minutes with 4,959 of 5,000 GraphQL points unspent, because several jobs sent calls at the same time.

So every call that reaches GitHub is also written to one ledger per machine, rate.log (epoch, resource, fg or bg). A call waits while the last BOB_FORGE_RATE_WINDOW seconds already hold the resource’s ceiling. The ledger is read and written under a lock directory, so two callers never both take the last slot.

Caller	May use	Over it
Foreground (any caller by default)	the whole ceiling	held until the window has room (hold in budget.log)
Background (BOB_FORGE_BACKGROUND=1)	BOB_FORGE_BACKGROUND_SHARE percent of the ceiling	deferred until the window drains (defer), or refused with exit 75 when BOB_FORGE_BACKGROUND_DEFER=0 (refuse)

A foreground call in the same window still goes through while a background call waits. bob brief reconcile and completions, and bob tick’s briefing step, run as background. A deferred call prints one line:

[forge-budget] background call deferred 34s: graphql rate is 31/60s, above the background share of 30 (ceiling 60). It runs on its own.

bob forge health prints the rate beside the quota:

rate: graphql 31 of 60 (12 background) — background calls deferred in the last 60s; background share 50%

Backoff

The gate cannot see a burst limit, and another machine can spend the quota between the read and the call. So a rate-limited answer is retried inside the shim, before the cache or the queue is reached:

Answer	Wait before the retry	Bound on the total wait
Names a Retry-After	that many seconds	BOB_FORGE_BACKOFF_MAX
A burst limit (secondary rate limit, abuse detection, HTTP 429)	5 s, then doubling (BOB_FORGE_BACKOFF_START)	BOB_FORGE_BACKOFF_MAX
The quota is spent (the primary limit)	until the reset on file; doubling when no reset is known	BOB_FORGE_MAX_WAIT

The stderr of a retried attempt is dropped. The caller sees a slow call and one [forge-budget] … retrying in Ns line per retry, never the rate-limit message. Every rate-limited answer is still recorded as an observation, and every wait in budget.log. Only when the bound is spent does the call fall through: a read to the cache, a write to the queue, anything else to its real failure.

Three calls are never retried, because a retry re-runs the command line:

• a call that read its own stdin (--input -, @-) — the input is gone. A write whose body came through --body-file - is retried: the shim captured the body before the first attempt;
• a --paginate stream — its first pages may already be out;
• an interactive call.

Settings

Variable	Default	Meaning
BOB_FORGE_BATCH	1	0 reads one issue per call where a batch would be used.
BOB_GROOM_RETRY_MAX_WAIT	3900	Most seconds bob groom waits for a reset before its one retry.
BOB_FORGE_BUDGET	1	0 turns the gate and the backoff off. The test runner and the operator sandbox set 0: a suite drives a fake gh.
BOB_FORGE_BUDGET_TTL	60	Seconds a read of the quota is reused.
BOB_FORGE_FLOOR_GRAPHQL / _CORE / _SEARCH	300 / 200 / 3	Under this, an unattended call waits for the reset.
BOB_FORGE_PACE_GRAPHQL / _CORE / _SEARCH	1000 / 600 / 8	Under this, an unattended call is slowed.
BOB_FORGE_PACE_SECONDS	2	The pacing sleep.
BOB_FORGE_MAX_WAIT	3900	Most seconds one call waits for a reset (a full window plus five minutes).
BOB_FORGE_BACKOFF_START / BOB_FORGE_BACKOFF_MAX	5 / 300	First backoff wait, and the most seconds one call spends backing off.
BOB_FORGE_INTERACTIVE	unset	1 or 0 overrides the terminal test.
BOB_FORGE_RATE_GRAPHQL / _CORE / _SEARCH	60 / 120 / 20	Most calls per window across every caller on the machine. 0 turns per-minute pacing off for that resource.
BOB_FORGE_RATE_WINDOW	60	The pacing window, in seconds.
BOB_FORGE_BACKGROUND	0	1 marks the caller as background.
BOB_FORGE_BACKGROUND_SHARE	50	Percent of the ceiling a background caller may use.
BOB_FORGE_BACKGROUND_DEFER	1	0 refuses a background call over its share (exit 75) instead of waiting.

budget.log is TSV: epoch, event (refresh, refresh-unreadable, pace, wait, wait-capped, backoff, backoff-capped, hold, defer, refuse), quota, remaining (the window’s count for the last three), seconds, subcommand.

The quota belongs to a token. Machines that share a token share the quota, and each reads the same server-side number, so the file needs no syncing between them.

Callers never give up on a limit

The shim makes a rate limit a slow call. The callers are written so that a slow call, or a rate-limit message that reached them anyway, never ends a run early.

Caller	What it does
/groom and /warp-drive (the prompts)	State the rule: a slow gh call or a [forge-budget] line is a wait, let it finish; run a rate-limited command again; never stop, skip an item or end the session over a limit or a reset time read in some output.
bob groom (scripts/groom/groom.sh)	Carries the same rule in its headless prompt, and names its calls groom. A run that ends without its GROOM-RESULT: line and mentions a rate limit is run once more, after the quota resets (the reset on file plus 5 s, 60 s when none is on file, capped by BOB_GROOM_RETRY_MAX_WAIT, default 3900). A run always ends with a trailer: when the model gave none, the wrapper prints its own, … incomplete=1 reason=<rate-limit|no-result-line|timeout|claude-failed>, with result null and a non-zero exit.
bob brief (scripts/brief/)	Runs every gh call through scripts/forge/patient.js: the call is named brief, its timeout makes room for the wait the shim is about to make, and a rate-limited answer, or a timeout while the shim was holding the call, is run once more with room for a full backoff.

A script that runs gh with a short timeout undoes the shim’s wait: the shim is killed mid-wait and the script records a gap. patient.js is the runner for that case:

const patient = require('../forge/patient');
const r = await patient.run(execGh, args, { timeout: 20000, caller: 'brief' });
// r.attempts is 1 or 2; r.retried is true when it was run again

Fewer calls spent

Pacing and backoff handle a quota that is running low. The cheaper fix is to spend less. The call sites were measured on 2026-09-27 with budget.js measure, against the live forge, on this repo:

Call site	Command	gh calls	Wall time
Work discovery	discover-queue.sh --json	1	2.8 s
Brief collectors	brief.js --json	22	16.0 s
Flight-plan status	flightplan.js status <plan> (47 items)	38	38.0 s
Flight-plan remaining	flightplan.js remaining <plan> --json	38	39.2 s
Stale todo sweep	todo-consume.js stale	122	188.1 s

Discovery is one gh issue list. The brief collectors already read over REST, one request per registered repo (15), plus one GraphQL query and two searches. The flight plan and the stale sweep read one issue per call.

The flight plan now reads its items together:

Command	Before	After
flightplan.js status <plan>	38 calls, 38.0 s	3 calls, 3.9 s
flightplan.js remaining <plan> --json	38 calls, 39.2 s	3 calls, 4.1 s

scripts/forge/batch.js reads up to 40 issues in one GraphQL query, each under its own alias, in the shape gh issue view --json gives:

const { readIssues } = require('../forge/batch');
const { issues, missing, unread } = readIssues([3129, 2945, 2944]);

• issues is a Map of number to {number, title, state, body, labels, isPullRequest}.
• missing lists the numbers the forge has no issue for.
• unread lists the numbers whose batch failed. The caller reads those one by one, so a batch that fails costs a retry and never an answer.

flightplan.js keeps what it read for the life of the process, and its single-issue readers look there first. BOB_FORGE_BATCH=0 turns batching off. The output of status and remaining is byte-identical either way.

The stale todo sweep reads its links the same way (#3219). A batch that asked for comments would cost more points than the calls it saves, so the batch reads state, closedAt and body only. A link’s comments are read one call each, and only when the verdict turns on them: the link closed after the todo was filed and its body does not mention the todo.

Command	Before	After
todo-consume.js stale (bigbrain, 2026-10-09)	50 calls, 84.9 s	9 calls, 12.7 s

BOB_FORGE_BATCH=0 turns batching off here too, and the sweep prints the same report and labels the same todos either way.

The busiest caller on farm-01 over an hour on 2026-10-09 was the bigbrain dashboard server: 459 of the machine’s 1,081 calls, about 9 a minute with no tab open. Its notices build runs every 60 s and made two searches plus one gh issue view per “Decision Needed” issue. The server now keeps those comments across builds and reads an issue again only when the search row’s updatedAt has moved or the read is older than DECISION_COMMENTS_TTL_S (600 s, never below the refresh interval; DASH_NOTICES_COMMENTS_TTL_S overrides it). A new comment moves updatedAt, so an answer clears its notice on the next build.

Command	Before	After
Five notices builds, one minute apart (8 decision issues)	50 calls, 25.2 s	18 calls, 26.7 s

To measure a command:

node scripts/forge/budget.js measure -- node scripts/warp-drive/flightplan.js status 3198
node scripts/forge/budget.js measure --json -- bob brief --json

It runs the command for real, with a recorder in front of gh on PATH, and reads the quota before and after (bob-forge-measure/1). The calls are counted exactly. The points are the token’s spend between the two reads, so calls from other sessions in the same seconds are included: measure on a quiet token.

On the measured token the quota counter moved far less than one point per call: 1 to 4 GraphQL points across 38 to 122 calls, and none for the REST and search calls. Whether GitHub bills these reads below one point or reports used late was not established. The call counts and wall times above are exact, and the burst limits count requests, so fewer calls is the measure that was optimised.

What is recorded

The target is that no call is ever rate-limited. Whether that holds is measured, from four files in $BOB_HOME/forge/:

File	One line per	Columns
observations.log	gh call	…, quota, cost, caller, session
quota.log	read of the quota	epoch, then used / remaining / reset for graphql and for core
ratelimit.log	rate-limited answer	epoch, quota, subcommand, caller, session, primary or secondary
budget.log	refresh, pace, wait or backoff	epoch, event, quota, remaining, seconds, subcommand

• Caller is the directory the call was made from, plus /<kind> when the caller names itself with BOB_FORGE_CALLER (groom, brief, tick).
• Session is the first eight characters of CLAUDE_CODE_SESSION_ID, or -.
• Cost is 1 for a call that spends a quota and 0 for one that spends none. It is a floor: a command that pages costs more. gh does not report what a call cost, so the cost per call is nominal.
• Measured spend comes from quota.log: the rise in used between two reads, or what the new window had used when the quota reset between them. It is what the token spent, on every machine that holds it.

scripts/forge/budget.js trims each file when it reads it (3,000 lines; observations.log 5,000).

Seeing the quota

budget.js turns the files into one document, bob-forge-quota/1. Every surface below shows that document. None of them calls the forge to do so.

Surface	What it shows
bob forge health	Under the verdict: each quota (left, limit, reset, measured spend this hour), how fresh the reading is, which callers and sessions made this machine’s calls in the last hour, what callers were made to wait, and the rate-limited answers this week. --json carries it as quota.
bob doctor	One forge quota: line. INFO normally. WARN when a quota is under its floor, or when any answer was rate-limited this week. Never graded.
Fleet snapshot (audit.js snapshot / view)	summary.quota per machine, in compact form, and a “GitHub quota” table in the view.
Fleet feed and bob viz	quota on each machine row. /bob draws one line under the header from the freshest reading, with callers and rate-limited answers summed across machines.

$ bob forge health
forge: operational (recent-calls-succeeded) — 10/10 recent calls ok, source: observations
quota: graphql 4,120 of 5,000 left, resets 10:47 (in 25 min), 1,480 spent in the last 60 min by this token
quota: core 4,990 of 5,000 left, resets 11:02 (in 40 min), 13 spent in the last 60 min by this token
quota: read 20s ago; floor graphql 300, core 200
spent by: 7 call(s) from this machine in the last 60 min
  bigbrain (session 56b20441): 3 call(s), at least graphql 2, core 1
  example-web/groom: 2 call(s), at least graphql 2, 1 rate-limited
held back: 1 wait(s) for a reset (5 min), 2 paced call(s), 1 retry wait(s) (5s) in the last 60 min
rate-limited answers this week: 2 (target 0) — last 2026-09-27T09:41:40.000Z on graphql from example-web/groom (primary)
  DEFECT: the forge is meant to prevent every one of these. File it as a bug on the forge (label bug, area:forge-resilience) with this output.

The quota is a reading beside the verdict. It never changes the state or the exit code: a forge that answers is operational however little quota is left. A rate-limited answer is a forge defect, because the budget exists to prevent it. It is reported as one wherever the quota is shown, for seven days.

The state of a quota is one of ok, pacing (under the pace line), waiting (under the floor) and reset (the reset time has passed since the read, so the number is the last window’s).

Verification: make test-forge-budget drives the shim with a fake gh and a fake sleep, so the suite waits for nothing. make test-forge-quota covers the document, every surface that shows it, and the patient runner. make test-forge-batch covers the batched read and the flight plan’s use of it. make test-bob-groom covers groom’s retry and its trailer.

Consumers

Surface	What it does with the signal
Stale todo sweep (todo-consume.js stale)	Prints UNCHECKED #NN for todos whose refs could not be read; the summary reports how many could not be fully checked instead of asserting every link is open.
/what-next	Step 0 checks the condition before any gh read, leads the report with it when impaired, and never reports “no work found” while the forge is not operational. Also gates the health verdict — idle and “could not read the queue” are the same empty list.
Fleet snapshot (audit.js snapshot / view)	Carries summary.forge per machine and renders a Forge section, including cached reads next to an otherwise-clean verdict. Deliberately per-machine: network path and credentials differ per host, so “the forge is up” is not a fleet-wide fact.
The read cache (bin/gh → scripts/forge/cache.js)	Falls back to last-known state on an availability failure, so a run survives a degradation instead of dying mid-phase.
The write queue (bin/gh → scripts/forge/queue.js)	Spools a mutation it cannot deliver and replays it in order on recovery, so a degraded run loses no PM state. Uses the same net/server/ratelimit/auth boundary the cache does.
Warp-drive degraded mode (scripts/warp-drive/forge-mode.js, #1751)	Every transition folds the verdict into state.forge: degraded/unreachable enters degraded mode (coding, testing and committing carry on over cached reads and spooled writes), entering merging parks the run with the reason (a plain git push never parks on a rate limit, #3191), the engine replays the write queue at every item boundary, and state-machine.js forge-check probes and resumes it once operational. Reads the observation log only (probe: false) at transitions — forge-check is the one place it probes. See warp-drive → Forge degraded mode.

New consumers should ask isTrustworthy() rather than re-deriving the rule. The question is always the same one — may this empty result be believed?

Related

• gh Adapter Contract — the gh surface this shim carries, read from the tree (#1753).
• Mirror Remote — the same resilience family, at the git layer.
• TODO Format Contract — the stale sweep, forge health’s first consumer: it now prints UNCHECKED #NN for todos whose refs could not be read, and its closing line reports what it could not establish rather than asserting every link is open.