CI Locus

Where a project’s GitHub Actions jobs run — on a BoB fleet self-hosted runner or on GitHub-hosted ubuntu-latest — is a declared project setting with exactly one source of truth, reconciled into GitHub rather than hand-set there (#1728).

Before this, the choice lived only as the repo variable CI_RUNNER_LABEL, which every workflow reads as runs-on: ${{ vars.CI_RUNNER_LABEL || 'ubuntu-latest' }} (#1694, #1782). Nothing declared it, validated it, or noticed when it drifted — the deploy target held the truth, the Prime Directive inverted. The runner runbook’s runner.sh target could flip the variable; what was missing was the declaration it should flip from.

The declaration

{
  "_meta": {
    "project": "bigbrain",
    "ci": { "locus": "farm" }
  }
}

Field	Values	Meaning
`locus`	`farm` \| `hosted`	Required. `farm`: CI runs on a fleet self-hosted runner. `hosted`: GitHub-hosted runners.
`runner_label`	a runner label (`[A-Za-z0-9][A-Za-z0-9._-]*`)	Optional, `farm` only. The label to target instead of the default `self-hosted-bob` (`runner.sh`’s `DEFAULT_LABEL`).

The block lives in the project’s provision manifest, provisions/<project>.json under _meta.ci, and is validated by schemas/manifest.schema.json (make check-schemas; additionalProperties: false, so a typo is refused rather than ignored).

Why the manifest and not dev.json. The manifest is already the declared, schema-validated, operator-owned (#2270) per-project settings home that bob provision and the fleet snapshot read from any machine through operatorPath(). dev.json is per-checkout dev-environment state: it is absent on CLI and tooling repos (this one included), it is regenerated by bob ready (#2463), and it describes one checkout — while a CI locus is a fact about the repo. The decision is recorded on #1728.

Declare intent, not a registration. farm names where CI should run, never a specific runner registration. GitHub runner groups are organization-only, so today’s per-repo registration is a consequence of the account shape, not a design choice (#1725); when a shared pool arrives, what farm projects to changes, not the schema.

Resolution

scripts/fleet/ci-locus.js is the one resolver. resolve() walks:

Rung	Read	`source`
1	`provisions/<project>.json` → `_meta.ci`	`project`
2	`provisions/_default.json` → `_meta.ci` (the account default)	`default`
3	nothing declared anywhere → `hosted`	`undeclared`
—	a block that is present but wrong (unknown locus, extra key, bad label, unparseable manifest) → `hosted`	`invalid`

Every result carries reason — which manifest declared what, or why it fell through — so a consumer can say why a project resolved the way it did, not only what it resolved to. undeclared and invalid are distinct: absence is a reported state, an error is an error, and neither is silently converged either way (below).

node ~/.claude/scripts/fleet/ci-locus.js resolve --project bigbrain
# bigbrain: farm (self-hosted-bob) — declared in provisions/bigbrain.json _meta.ci
#   provisions/bigbrain.json _meta.ci declares locus farm
#   projection: CI_RUNNER_LABEL = self-hosted-bob (runs-on self-hosted-bob)

resolve reads disk only — it never calls gh.

Projection

The GitHub side is a projection of the declaration, never the truth:

Declared	Repo variable `CI_RUNNER_LABEL`	Every workflow’s `runs-on`
`farm`	set to `runner_label`, else `self-hosted-bob`	that label
`hosted`	deleted	`ubuntu-latest`

An unset variable evaluates to the empty string, so unset is the hosted default — there is no separate “off” state that can disagree with it (#1694). That is why hosted deletes rather than writes.

Drift report — `status`

status reads what GitHub holds now (the variable, whether the repo is private, and how many online runners carry the expected label) and compares:

`verdict`	Meaning	Exit
`converged`	the variable matches the declaration’s projection	0
`drift`	it does not — the output names both sides and prints the exact reconcile command as `fix`	1
`undeclared`	no declaration — hosted by absence; the actual is shown but nothing is judged	0
`invalid`	the declaration is wrong — fix the manifest	1
`unmeasured`	the repo side could not be read (no origin remote and no `--repo`, `gh` down, or the probe is off) — never a guess	4

Two conditions ride alongside the verdict, never folded into it:

degraded — declared farm but no online runner carries the label: the queue-forever state runner.sh status already exits 3 on. Converged and degraded at once is a real (and common) state.
hazard — the repo is public and declared farm. A self-hosted runner on a public repo lets any fork pull request execute arbitrary code on the fleet (Security constraints). A hazard wins the exit code (3) whatever the verdict.

node ~/.claude/scripts/fleet/ci-locus.js status --project bigbrain [--repo owner/name] [--no-probe] [--json]

--json prints one bob-ci-locus/1 document. BOB_CI_LOCUS_PROBE=0 turns every GitHub read off (every consumer then reports not measured) — what the offline test suites and a forge-down snapshot use.

Reconcile

node ~/.claude/scripts/fleet/ci-locus.js reconcile --project <name> [--repo owner/name]          # dry run
node ~/.claude/scripts/fleet/ci-locus.js reconcile --project <name> [--repo owner/name] --apply  # write

Reconcile reads the repo side first and refuses, in this order, before any write:

Condition	Outcome	Exit
repo side unreadable	error — nothing written	4
`undeclared`	left exactly as GitHub holds it — absence is never converged either way; declare `_meta.ci.locus` to converge	0
`invalid`	refused, naming the manifest to fix	4
public repo declared `farm`	refused, the hazard named	3
`converged`	no-op — no GitHub write	0
`drift`, dry run	`would-change`, printing the exact `runner.sh target …` command	1
`drift`, `--apply`	runs `scripts/fleet/runner.sh target <label\|hosted> --repo <slug>`, then re-reads the repo to verify: `applied` (0) or `applied-unverified` (4)	0 / 4
`runner.sh` refuses (no online runner carries the label)	`failed`, its message surfaced verbatim, nothing changed — never forced past	4

Two properties follow. Idempotent: a second --apply on a converged repo performs no GitHub write at all — the test suite pins “exactly one set on drift, none on the re-run”. The public-repo guard is enforced at reconcile time, not only at runner-install time, so a repo that flipped public after its runner was registered is caught on the next reconcile even though the variable already matches (runner.sh install refuses a public repo too; this closes the window after it).

runner.sh target stays the only writer of CI_RUNNER_LABEL: reconcile composes it rather than calling gh variable itself, so the runner-online guard and the hosted-means-unset rule live in one place. A temporary flip (runner.sh target hosted during a maintenance window — runbook step 5) is legitimate and shows as drift until it is reverted or the declaration is changed; that visibility is the point.

Runner-availability preflight (#1731)

A self-hosted runner that is offline does not fail jobs — it parks them. The job sits Queued, the check never reports, and anything waiting on it waits forever: warp-drive’s merge gate blocks on a check that will never arrive, and the operator sees a session that looks busy rather than broken. Hosted Actions fails in three seconds; the farm fails in silence. The preflight makes the farm fail loudly, or not at all:

node ~/.claude/scripts/fleet/ci-locus.js preflight --project <name> [--repo owner/name]                  # dry run
node ~/.claude/scripts/fleet/ci-locus.js preflight --project <name> [--repo owner/name] --apply          # act
node ~/.claude/scripts/fleet/ci-locus.js preflight --project <name> --no-fallback --apply                 # never route to hosted

It asks one question — will the next job run? The next job runs where the repo variable points now (the effective target), so the runner probe is taken against that through runner.sh status --json (whose exit 3 is exactly the queue-forever condition), and the declaration decides what to do about a miss:

Effective target	Runners	Fallback policy	Hosted	Verdict	Exit
a label	an online runner carries it	—	not probed	`ok` — the next job runs on the fleet	0
a label	none online	`none`	not probed	`degraded` — the operator is told the exact `runner.sh target hosted` command	3
a label	none online	`hosted`	viable	dry run `would-fallback` (1); `--apply` `fallback`: `CI_RUNNER_LABEL` cleared through `runner.sh target hosted`, the next job runs on `ubuntu-latest` — within one job, never after a queue timeout, and reported	1 / 0
a label	none online	`hosted`	dead	`no-viable-runner` — one report naming both failures, nothing written	3
hosted	declared `farm` and an online runner carries its label	—	not probed	dry run `would-restore` (1); `--apply` `restored`: the declared locus is back, no manual re-flip	1 / 0
hosted	declared `farm`, an online runner carries its label, the repo is public	—	viable	`ok` on the fallback with `hazard` named — the restore is refused on the dry run and under `--apply`, nothing written (#3657)	0
hosted	declared `farm`, none online	—	viable	`ok` on the fallback (`fallback.active: true`), restored once a runner is back	0
hosted	declared `farm`, none online	—	dead	`no-viable-runner` naming both	3
hosted	declared `hosted` / undeclared	—	viable / dead	`ok` / `hosted-dead`	0 / 3
—	`gh` down, probe off, no origin remote	—	—	`unmeasured` — never a guess, nothing written	4

The fallback policy is declared next to the locus — _meta.ci.fallback: "hosted" (the default) or "none" — and overridden for one run by --no-fallback or BOB_CI_FALLBACK=0. A fallback is a visible GitHub write through the one writer (runner.sh target hosted), so it shows as [drift] in bob provision status until the preflight restores the declared locus; that visibility is the point. reconcile --apply would refuse to move the variable back while no runner is online, so the two verbs never fight.

Hosted viability is judged from evidence, never assumed: the newest completed workflow run in the last 20 and its jobs. A job that concluded failure with zero steps and no runner was never started — the billing / spending-limit outage of todo #1648 looks exactly like that — and a run concluded startup_failure is the same signal. No completed run to judge from reads as unknown (reported) and the fallback proceeds; a dead hosted is never fallen back onto, and the no-viable-runner report names the offline runner and the never-started job, its run and URL, in one line. Hosted is only probed when the decision depends on it (the farm runner is offline, or CI already targets hosted). How close an account is to that outage before it happens is the Actions budget probe (#1695).

--json prints one bob-ci-preflight/1 document: {schema, project, slug, apply, declared{locus, label, source, declared, reason, online_with_label}, fallback{policy, source, active}, effective{runs_on, variable, locus, online_with_label}, verdict, cause, fix, action, wrote, farm{checked, exit, ci_targets, variable, private, runners[], degraded, error}, hosted{checked, viable, reason, run, job, error}, hazard, exit}. The exit code is the answer: 0 the next job runs (ok / fallback applied / restored), 1 would-change (dry run), 3 blocked with the cause named (no-viable-runner, hosted-dead, degraded with fallback off, a refused write, an invalid declaration), 4 not measured.

Recovery is idempotent

When the runner comes back the next preflight returns CI to the declared locus on its own: declared farm, an online runner carrying the label, and CI_RUNNER_LABEL unset (a fallback still in place) is would-restore on a dry run and restored under --apply — one set of the variable through runner.sh target <label>, no manual re-flip. A second pass is ok with no write. The whole outage cycle is ok → fallback → ok → restored → ok with exactly two GitHub writes (delete, set), pinned by the test suite. The restore is only ever to the declaration: a repo declared hosted is never “restored” to a label because a runner happens to be online.

The restore never points a public repo at the fleet (#3657). The probe carries the repo’s visibility (farm.private), and a declared-farm repo that is public is never would-restore or restored, however many runners carry its label: the preflight stays ok on the hosted fallback, writes nothing, names the hazard on the document (hazard, and a hazard: line in the human output) and gives the way out in fix — declare hosted or make the repo private. runner.sh target <label> refuses the same write on its own side, with no override, so the guard holds even when the preflight is bypassed by hand; target hosted stays allowed on a public repo because it is the way out of the hazard. A visibility gh cannot read is refused by target too — a repo is pointed at the fleet only when it is known private.

Queued-job alarm (#1731)

A job parked past a declared threshold is the silent stall made visible — and the alarm says which kind, because “queued” has two very different causes:

node ~/.claude/scripts/fleet/ci-locus.js queue --project <name> [--repo owner/name] [--threshold <min>] [--json]

Kind	Condition	Alarm	Exit
`clear`	nothing queued past the threshold	no	0
`busy`	queued past the threshold, every online runner with the label is busy — queued because busy: the queue drains as jobs finish	no	0
`offline`	queued past the threshold and no online runner carries the label — queued because offline: nothing will ever pick the job up	yes	3
`idle-runner`	queued past the threshold although an online runner with the label is idle — a label mismatch on the job, a runner-group restriction or a stuck runner service	yes	3
`hosted`	CI targets `ubuntu-latest` and a run is parked — GitHub-side capacity (a billing outage shows as failed-with-zero-steps, not queued)	yes	3
—	`gh` down, probe off, no origin remote	never invented	4

The threshold is declared per project — _meta.ci.queue_alarm_minutes (an integer ≥ 1, default 10) — and overridden for one call with --threshold. Age is measured from each run’s created_at; the runner list is read once through runner.sh status (or reused when the preflight already holds it). Every alarm carries a fix: offline names the preflight (preflight --apply falls back to hosted when it is viable) and the gh run rerun for the oldest parked run; idle-runner names the gh run view … --json jobs comparison against runner.sh status.

--json prints one bob-ci-queue/1 document: {schema, project, slug, measured, threshold_minutes, threshold_source, effective{runs_on, variable}, queued[{id, name, branch, created_at, age_minutes, url}], over_threshold, oldest_minutes, runners{online, busy, idle, with_label}, kind, alarm, cause, fix, error, declared, exit}.

warp-drive runs the preflight itself (#1731 AC-02): on entry to the merging phase — before the run pushes anything CI must grade — the state machine calls preflight for a project that declares _meta.ci (--apply at Level 3+, a dry run at Level 2 that hands the human the command; _workflow.ci_runner_preflight: false skips it, an undeclared project is skipped without a network call), records the verdict on state.ci_preflight, and appends a CI RUNNER PREFLIGHT block to the phase instruction. A blocked verdict is a named blockable event: the run files a conforming todo with the cause and fix, fires merge_failed with ci_verdict set, and escalates — it never opens the PR and waits out a merge-on-green timeout. A fallback or restore the preflight performed is reported in the chunk report and the session summary’s CI runner line, never silent.

Where the alarm surfaces. The preflight carries the same record as queue on its own document (queue: null only with queue: false), so one read feeds both. The fleet snapshot records it per declared, measured project as ci.queue — {measured, alarm, kind, queued, over_threshold, oldest_minutes, threshold_minutes, cause, fix, error}, null when the project was not probed — rolls it up as summary.ci.queue_alarm and summary.ci.queue_alarm_names[] (name (kind Nm)), and audit.js view adds a QUEUED column to the CI locus table with the parked projects marked … QUEUED and, when any fired, one line explaining the kinds. bob warp status prints a CI runner: line from the preflight the engine recorded on entry to merging (state.ci_preflight, exposed as ci_runner by state-machine.js status): the verdict and target, (hosted fallback active) when one is in place, the cause, and QUEUE ALARM (<kind>): N run(s) past Tm, oldest Nm in red — queue clear / queue busy otherwise, not checked before the run has reached merging.

Where it surfaces

bob provision status prints a CI locus (#1728) block after the item types, through the same resolver:

CI locus (#1728) — paulirv/bigbrain:
  declared: farm (self-hosted-bob) — declared in provisions/bigbrain.json _meta.ci
  actual:   CI_RUNNER_LABEL=self-hosted-bob → runs-on self-hosted-bob (private repo, 1 online runner with label self-hosted-bob)
  [converged] declaration and GitHub agree

The verdict line is one of [converged], [drift] (both sides and the reconcile command), [undeclared], [invalid] or [not measured], with [hazard] and [degraded] lines when they apply. The GitHub read is best-effort: a checkout with no origin remote, no gh, or BOB_CI_LOCUS_PROBE=0 reports not measured and the command stays offline. The block never changes bob provision status’s exit code.

The fleet snapshot (scripts/fleet/audit.js snapshot, #415) carries per present project:

"ci": { "locus": "farm", "label": "self-hosted-bob", "runs_on": "self-hosted-bob",
        "source": "project", "declared": true, "reason": "…", "slug": "paulirv/bigbrain",
        "measured": true, "verdict": "converged", "drift": false,
        "actual_variable": "self-hosted-bob", "private": true, "degraded": false,
        "hazard": null, "fix": null,
        "queue": { "measured": true, "alarm": false, "kind": "clear", "queued": 0, "over_threshold": 0,
                   "oldest_minutes": null, "threshold_minutes": 10, "cause": "nothing queued", "fix": null, "error": null } }

Resolution is disk-only and always recorded. The drift probe is one gh read per declared project, taken only while the forge is operational (summary.forge) and BOB_FLEET_CI_PROBE is not 0; an undeclared or invalid project is never probed (nothing to converge, nothing to measure), and an unprobed declared project reads verdict: unmeasured, drift: null. The machine roll-up is summary.ci — {projects, declared, farm, hosted, undeclared, invalid, measured, drift, hazard, degraded, queue_alarm, queue_alarm_names[], names[], hazard_names[], by_project} — and audit.js view renders a CI locus table with MACHINE / DECLARED / FARM / HOSTED / UNDECL / MEASURED / DRIFT / HAZARD / QUEUED / PROJECTS rows per machine and a FLEET row that counts each project once, preferring the machine that measured it against the repo. null on either means the resolver was absent, never “no drift”.

Tests

tests/test-ci-locus.js — the schema accept/reject set (including fallback), every resolver rung and reason, every status verdict, the reconcile refusal order (exactly one write on drift, none on the re-run; delete for hosted; the public-repo refusal drifted and converged; undeclared left alone in both directions; the runner.sh guard), the bob provision status block and every preflight verdict (#1731: ok, would-fallback / fallback with exactly one delete and an idempotent re-run, degraded under each fallback-off route, no-viable-runner from a never-started job and from startup_failure with the variable untouched, hosted-dead, unmeasured; the restore leg and the full ok → fallback → ok → restored → ok cycle with exactly two writes; every queue kind, the declared threshold and --threshold), against a self-contained fixture BOB_SOURCE and a fake gh that serves a state file and records every write. tests/test-fleet-audit.js pins the snapshot record (with its queue field and the alarm from a parked run), the roll-up, the never-probe-undeclared rule and the view table; tests/test-warp-status-preflight.sh pins the CI runner: line of bob warp status.

Job isolation between pull-request and release jobs (#3656)

A farm locus means one persistent runner takes every job of a repo — a pull-request’s checks and, later, its release and deploy jobs. Between jobs the runner kept _work/_tool (setup-node’s toolchains) and _work/_actions (the action repos), so a pull-request job could tamper with code a later release job would run, and ci.yml’s isolation step only warned when a runner was not isolated at all.

The choice: one label, two job hooks — not a second label. A separate label for release and deploy jobs would need a second runner registration on every machine and a second runs-on seam in every workflow, which the workflow-authoring gate exists to prevent. Instead runner.sh hygiene installs scripts/fleet/runner-job-hook.sh under two names and the name tells the hook its phase: bob-job-hook.sh (the job-started hook) keeps _work/_tool and _work/_actions — this job’s own downloads; bob-job-hook-completed.sh (the job-completed hook) empties them too, so nothing a job left behind waits for the next one. The cost is a re-download of the toolchain and the actions per job, seconds on the farm. ci.yml’s Verify job isolation step now fails a job on a runner that is not isolated (BOB_RUNNER_ISOLATED unset) instead of warning. The mechanism and the hygiene command are in the runbook’s Workspace isolation; tests/test-fleet-runner.sh runs both hooks and the step the way the runner and CI run them.

CI Locus

CI Locus

The declaration

Resolution

Projection

Drift report — status

Reconcile

Runner-availability preflight (#1731)

Recovery is idempotent

Queued-job alarm (#1731)

Where it surfaces

Tests

Job isolation between pull-request and release jobs (#3656)

See also

Drift report — `status`