Runbook: Self-Hosted Actions Runner

Purpose: Run CI on fleet hardware instead of GitHub-hosted runners, so the job bills zero Actions minutes — and get back to hosted safely when the machine is unavailable When to use: Included Actions minutes are the binding constraint (#1648, #1690), or you want CI executing on real fleet hardware Prerequisites: The repo is private (non-negotiable — see Security constraints); admin on the repo; gh authenticated (gh auth status); the target machine reachable over Tailscale SSH and listed in machines.json, with the account its runners run as declared there (ci_runner.user — see The runner’s account); curl, tar, rsync, git, gh, jq, make and tmux on the target (the per-workflow audit says which workflow needs which; make test drives a real tmux server since #2043); sudo on the target for the service step Estimated time: 10 minutes, plus one CI run to confirm

Everything here goes through one script, scripts/fleet/runner.sh. Each mutation has an inverse, and none of it is hand-edited state:

Command	What it does
`runner.sh status`	Who is registered, is it online, what do the workflows currently target
`runner.sh install --machine <id>`	Register the machine and start its runner service
`runner.sh target <label>`	Point the workflows at a runner label
`runner.sh target hosted`	Point them back at GitHub-hosted `ubuntu-latest`
`runner.sh remove --machine <id>`	Reset the CI target, stop the service, deregister

Every mode acts on this repo unless you pass --repo <owner/name>, which points it at any other repo you have admin on (#1814) — see Another repo.

The CI switch itself is a single repo variable, CI_RUNNER_LABEL, read by every workflow — ci.yml, docs-deploy.yml, and auto-release.yml, plus the two scaffolded templates they ship from (#1782):

runs-on: ${{ vars.CI_RUNNER_LABEL || 'ubuntu-latest' }}

An unset variable evaluates to the empty string, so unset is the hosted default — there is no separate on/off flag that can disagree with it.

The declaration is the thing to edit, not the variable (#1728). Which locus a project should run on is declared once, in its provision manifest — provisions/<project>.json → _meta.ci: { "locus": "farm" | "hosted" } — and the repo variable is a projection of it. scripts/fleet/ci-locus.js reconcile --apply sets or clears CI_RUNNER_LABEL through runner.sh target only when the two disagree, bob provision status and the fleet snapshot report [drift] when they do, and a repo with no declaration is reported undeclared rather than switched either way. runner.sh target remains the mechanism (and the right tool for a temporary flip — see step 5); the declaration is the record. Full reference: CI Locus.

One workflow honouring the switch is not enough, and the failure mode is quiet: a hosted job on a repo with no available Actions minutes fails in about three seconds with no steps and no runner assigned, which reads as an ordinary red X. docs-deploy and auto-release sat like that for weeks — the docs site published nothing and no release was cut — while ci.yml, already on the variable, stayed green and made the repo look healthy. make check-workflow-runs-on now refuses any shipped workflow or template whose job is not on the seam, and tests/test-fleet-runner.sh pins the six shipped files by name; a job that genuinely cannot run on the farm is pinned explicitly with # pinned: <reason> (#1729 — Workflow Authoring).

What a runner must carry. Node comes from actions/setup-node (no system Node needed), but the workflows assume the host provides jq, gh, git, rsync, tar, make and tmux. ci.yml asserts jq and tmux up front and docs-deploy.yml asserts jq, so a missing binary fails with a sentence rather than a parse error deep inside a build. The full per-workflow audit — every binary, how it reaches the runner, and the farm run that proved it — is below.

Steps

1. Check the current state

scripts/fleet/runner.sh status

Verify: the repo reports private: true, and the CI targets line names either ubuntu-latest (variable unset) or a label with an online runner behind it. Exit code 0 means consistent; exit 3 means CI is pointed at a label nothing is serving — jump to When the runner is offline.

If the repo is not private, stop. Do not register a runner.

2. Preview the registration

scripts/fleet/runner.sh install --machine farm-01 --print

This mints no token, touches nothing, and needs no network. It prints the three phases that install will run, so you can read exactly what lands on the machine before it lands:

Phase 0 (privileged, no secret): make sure the declared account exists. A missing one is created as a system account with no login shell (Linux). The phase refuses the account you are logged in as, and one that may sudo.
Phase A (unprivileged): as the runner’s account, download the runner tarball into actions-runner-bob in that account’s home, then config.sh --unattended --replace to register it. This is the phase that carries the registration token. It is streamed over ssh stdin and handed to config.sh through its environment (ACTIONS_RUNNER_INPUT_TOKEN), so the token never enters an argument vector on either machine.
Phase B (privileged): svc.sh install <account> + svc.sh start. The service runs as the runner’s account. The phase carries no secret, which is why it can safely be a separate invocation with a sudo tty.

Pin a specific runner release with RUNNER_VERSION if you need to; by default the target resolves the latest release itself.

3. Register the machine

scripts/fleet/runner.sh install --machine farm-01

Phases 0 and B prompt for the sudo password on the target. Phase A runs as the runner’s account through sudo -u; over ssh that needs sudo without a password on the target, so on a machine without it, run the command on the machine itself. Re-running the command is safe: --replace re-registers the same runner name rather than accumulating duplicates.

install exits 2 and does nothing when the machine declares no runner account, or declares root or the account you log in with — see The runner’s account.

Verify:

scripts/fleet/runner.sh status

The runner appears as farm-01 online [self-hosted-bob]. CI still targets ubuntu-latest at this point — registration and switching are deliberately separate, so a half-finished setup never redirects CI.

4. Point CI at the runner

Declare it, then reconcile — the declaration is what bob provision status and the fleet snapshot hold the repo to from now on:

# provisions/<project>.json → "_meta": { …, "ci": { "locus": "farm" } }
node scripts/fleet/ci-locus.js reconcile --apply      # runs runner.sh target self-hosted-bob when they differ

runner.sh target self-hosted-bob alone flips the variable without a record — the next bob provision status reports it as drift against an undeclared or hosted declaration.

The command refuses if no online runner carries that label (--force overrides, for the case where you know the machine is mid-boot).

Verify: push a commit or re-run the CI workflow, then open the run. The job header names the self-hosted runner rather than a GitHub-hosted image, and the run consumes no billable minutes. gh run list --limit 1 should show it completing, not sitting in queued.

5. Go back to hosted

Any time — a machine reboot, a maintenance window, or a red run you suspect is environmental:

scripts/fleet/runner.sh target hosted

The next run uses ubuntu-latest again. Nothing needs deregistering to do this. While the declaration still says farm, bob provision status and the fleet snapshot report the flip as [drift] — which is the point: a temporary override stays visible until it is either reverted (ci-locus.js reconcile --apply) or made the declared state ("locus": "hosted" in the manifest). Never edit the variable to silence the report.

6. Remove the runner entirely

scripts/fleet/runner.sh remove --machine farm-01

Order matters and the script enforces it: the CI target is reset first, then the service is stopped, then the runner is deregistered. Removing a runner while CI still points at its label is the one sequence that strands the repo.

Another repo

Add --repo <owner/name> to any command above and it acts on that repo instead of this one. Nothing else about the procedure changes — same phases, same guards, same inverse:

scripts/fleet/runner.sh status  --repo paulirv/peabod-www
scripts/fleet/runner.sh install --repo paulirv/peabod-www --machine farm-01
scripts/fleet/runner.sh target  self-hosted-bob --repo paulirv/peabod-www

Many repos at once. ci-rollout.js runs the same install and target steps for every private, active repo in the declared accounts. It declares each repo’s locus first, and dry-runs by default. Its verify step catches the workflow problem below, because a repo whose jobs never read the variable has no green run on the farm.

Patch the target repo’s workflows first. The switch is a repo variable that does nothing unless the workflows read it. A repo whose jobs still say runs-on: ubuntu-latest will keep going to hosted after you set the variable, with no error anywhere — the same silent failure #1782 describes. Convert every workflow in the repo, not just ci.yml:

runs-on: ${{ vars.CI_RUNNER_LABEL || 'ubuntu-latest' }}

Each repo gets its own runner home. A runner directory holds exactly one .runner config, so two repos cannot share one — registering a second repo into an occupied directory replaces the first runner rather than adding to it. The path is derived as actions-runner-<repo-name> in the runner account’s home, overridable with --dir. This repo is the one exception: its runner predates the convention and stays at actions-runner-bob, which runner.sh resolves automatically.

The service units are namespaced by repo already (actions.runner.<owner>-<repo>.<machine>.service), so several runners coexist on one machine:

systemctl list-units 'actions.runner.*'

What one machine serving several repos costs you. Each runner is a separate process holding a job slot, and they compete for the same cores, disk, and network as whatever else the machine does — on farm-01 that includes warp-drive and cdfork sessions. How many runners a machine may carry, and how much of it all CI together may take, is declared — see Capacity. A new repo past max_runners is refused until the declaration is raised.

Deploy workflows run on the same seam as the gate. Since #1782 the docs deploy and the release cut follow CI_RUNNER_LABEL like ci.yml does, and on this repo they have run on the farm since 2026-09-14 — the earlier recommendation to keep CD hosted (#1725) was made when hosted minutes were still available, and the operator declined to restore them (#1729). What makes that acceptable is that a deploy credential never lands on the runner’s disk: GitHub injects each secrets.* reference into the job’s environment, masked, for the life of the job — see Security constraints for the rule and the inventory entries that declare each credential’s home.

What each workflow needs from the runner (#1729)

The toolchain audit behind “nothing here is hosted-only”. Every binary a shipped workflow uses, how it reaches the runner, and whether that path works on a self-hosted Linux runner. Anything a hosted image would have supplied by assumption — brew, a preinstalled Node or Python, a setup-* action leaning on a hosted cache — is listed with the way it is provided instead.

Workflow	Job(s)	Needs	Provided by	Farm-01
`ci.yml`	`check-and-test`	Node 22	`actions/setup-node@v4` — populates the runner’s tool cache on first use, no system Node needed	works
		Python 3.12 + PyYAML	`actions/setup-python@v5` then `pip install pyyaml` — a non-externally-managed interpreter, so plain pip works; nothing from the host’s `/usr/bin/python3`	works
		`jq`, `tmux`	the host; asserted in the first step (`tmux` is `apt-get`-installed only when `RUNNER_ENVIRONMENT` is `github-hosted`)	present
		`make`, `git`, `bash`, `gh`	the host (`gh` for `check-report-labels`, which is skipped under `$CI` anyway)	present
`auto-release.yml`	`release`	Node 22	`actions/setup-node@v4`	works
		`git`, `gh`, `jq`	the host — `release.sh` reads run conclusions and issues through `gh`	present
`docs-deploy.yml`	`config`	`jq`	the host; asserted	present
	`doc-governance`, `gate`, `deploy`	Node 22.12	`actions/setup-node@v4`	works
	`gate`, `deploy`	`rsync`, `tar`	the host — the docs build and setup-node’s tool-cache extraction	present
	`deploy`	wrangler	`cloudflare/wrangler-action@v3` runs it through `npx` after setup-node — nothing global	works
	`deploy`, `notify-*`	`actions/cache`, `sticky-pull-request-comment`, `gh`	JS / composite actions run unchanged on a self-hosted Linux runner; `gh` from the host	works

Nothing in the table is brew-installed or read from a hosted-only path, so no provisioning step was needed on farm-01 — the host already carried every binary in the “the host” rows (command -v jq tmux make git gh rsync tar resolves each on 2026-09-16), and the setup-* actions supply the rest. A new workflow follows the same rule: provide a tool through a setup-* action or assert it in a first step — never assume the image (Workflow Authoring).

Verified green on the farm — all three, once each (#1729 AC-03). Every job below ran on runner farm-01 (label self-hosted-bob), read from the run’s job list (gh api repos/paulirv/bigbrain/actions/runs/<id>/jobs):

Workflow	Run	Trigger	Jobs on farm-01	Result
`ci.yml` (the gate)	35038576730	push to master, 2026-09-16	`check-and-test`	success
`auto-release.yml` (the release cut)	34812964247	weekly schedule, 2026-09-14	`release`	success
`docs-deploy.yml` (the docs deploy)	35036304146	push to master, 2026-09-15	`config`, `doc-governance`, `gate`, `deploy (vitepress)`, `deploy (starlight)`, `notify-recovery`	success — production publish

A green farm run is evidence the code works on that machine (the filesystem persists between jobs — see below); it is the release and publish path now, so treat a red one as an outage, not an experiment.

The runner’s account (#3655)

A job runs with everything its account has. So a runner never runs as the account you log in with: on the farm that account holds your GitHub login, the bridge secret, ssh keys and sudo, and one CI step could write a hook or an authorized key that outlives the job. Each machine declares one account for its runners, on its machines.json entry:

"ci_runner": { "user": "bob-ci", "max_runners": 3, "labels": ["self-hosted-bob"], … }

The account holds nothing but runner homes. runner.sh does the rest:

Command	What the declaration does
`install`	Exits 2 when no account is declared, when it is root, when it is the account you log in with, or when it may sudo (it is in `sudo`, `wheel`, `admin` or `root`). Otherwise it creates a missing account as a system account with no login shell, registers the runner as that account in that account’s home, and installs the service for it.
`status`	Prints `runs as: <account>` for each runner, read from the `User=` of its service unit, and flags a runner on the account you log in with: `<- the operator account`, plus a `SECURITY` block. `--json` carries `service_user` and `operator_account` per runner; both are `null` when the machine could not be reached.
`service`, `update`, `remove`, `capacity`, `hygiene`	Find the runner homes in the account’s home. Everything written inside a runner home is written as the account.

A machine that declares no account can still run those last five against the runners in the login user’s home. That is what lets you move a runner that was registered before the declaration existed.

Move a runner off the account you log in with

runner.sh status shows which runners need it. Do every runner on the machine in one sitting, because the declaration is per machine and remove finds a runner’s home from it:

# 1. While machines.json still declares no ci_runner.user: point CI at hosted,
#    stop the service and deregister. Repeat with --repo for each other repo.
scripts/fleet/runner.sh remove --machine farm-01

# 2. Declare the account: machines.json → farm-01 → "ci_runner": { "user": "bob-ci", … }

# 3. Register under the new account and point CI back. Repeat with --repo.
scripts/fleet/runner.sh install --machine farm-01 --target

# 4. Check.
scripts/fleet/runner.sh status          # … runs as: bob-ci

# 5. Once a job has passed on the new runner, delete the old home.
rm -rf ~/actions-runner-bob

Between steps 1 and 3 the repo’s CI targets hosted runners. The new account sees only system-wide tools (/usr/bin), not anything installed under your home, so the binaries in the per-workflow audit have to be installed system-wide.

Capacity: how many runners, and what CI may take (#1732)

A runner service executes one job at a time, so the number of runner services on a machine is its CI concurrency — and on farm-01 those jobs share four cores and 7 GiB with warp-drive, bob fork and the nightly tick. The capacity is declared once, on the machine’s entry in machines.json (operator data), and runner.sh provisions it:

"ci_runner": {
  "user": "bob-ci",
  "max_runners": 3,
  "labels": ["self-hosted-bob"],
  "reason": "why this many, and why this reserve",
  "reserve": { "cpu_quota": "200%", "memory_max": "3G", "nice": 10 }
}

Field	Enforced by
`user`	The account every runner on the machine runs as; `runner.sh install` refuses a machine without one (exit 2) — The runner’s account
`max_runners`	`runner.sh install` refuses a new runner home past it (exit 3), counting every `*/.runner` in the runner account’s home; re-registering an existing home never counts twice
`labels`	`runner.sh install --label` outside the list is refused (exit 2)
`reserve`	`/etc/systemd/system/bob-ci.slice` with `CPUQuota=` and `MemoryMax=` bounds all runners together; a drop-in (`<unit>.d/bob-capacity.conf`) places each runner service in the slice with `Nice=` so an agent session wins a contended core

Without a ci_runner block install behaves as before — no limit and no reserve. The schema is schemas/machines.schema.json → ci_runner.

scripts/fleet/runner.sh capacity --machine farm-01          # the declaration, and the privileged command --apply runs
scripts/fleet/runner.sh capacity --machine farm-01 --apply  # write the slice and drop-ins, restart idle runners into it

install applies the reserve itself after the service step; capacity --apply is for a changed declaration or a runner installed before it existed. Both are idempotent: a file is rewritten only when its content differs, and a second run reloads and restarts nothing. A runner in the middle of a job is never restarted — it is told it joins the slice on its next restart. Linux only: launchd has no slice, so on a Mac the reserve is reported as skipped.

Verify: systemctl status bob-ci.slice lists each actions.runner.*.service under the slice, and systemctl show -p Slice,Nice actions.runner.paulirv-bigbrain.farm-01.service prints Slice=bob-ci.slice and the declared Nice.

Workspace isolation: every job starts clean (#1732)

A hosted runner is a fresh VM per job; a fleet runner is not. Its _work directory, its temp dir and its HOME persist, so a run can pass only because of what the previous one left behind — a node_modules, a cache, a git config. A run like that is a defect, not a green. One command makes every runner home on a machine start each job clean:

scripts/fleet/runner.sh hygiene --machine farm-01          # the command --apply runs
scripts/fleet/runner.sh hygiene --machine farm-01 --apply  # write it, restart idle runners to read it

install runs it for you after the service step. Per runner home it installs:

What	Effect
`bob-job-hook.sh` and `bob-job-hook-completed.sh` (two copies of `scripts/fleet/runner-job-hook.sh`) as `ACTIONS_RUNNER_HOOK_JOB_STARTED` and `ACTIONS_RUNNER_HOOK_JOB_COMPLETED` in `.env`	Before the first step and after the last, empties the job’s `_work/<repo>` (the checkout and anything beside it), `RUNNER_TEMP` and the isolated HOME. The name tells the hook its phase (#3656): the started hook keeps `_work/_tool` and `_work/_actions` — this job’s own toolchain and action downloads; the completed hook empties them too, so tool or action code a pull-request job left behind never runs inside a later release or deploy job on the same runner. Deletes only inside its own runner home — anything else is refused and reported.
`HOME=<runner home>/_home`	A job’s `~` is an empty directory, emptied per job. Nothing one job left there — a `.gitconfig`, a `.npmrc`, a `gh` or wrangler login — reaches the next.
`GIT_SSH_COMMAND=ssh -F none -o IdentityFile=none -o IdentitiesOnly=yes -o IdentityAgent=none`	OpenSSH finds `~/.ssh` from the passwd entry, not `$HOME`, so moving HOME alone would still hand a job whatever keys its account holds. A job’s git-over-ssh offers no key.
`BOB_RUNNER_ISOLATED=1`	The marker `ci.yml`’s Verify job isolation step reads: on an isolated runner it fails the run if HOME is the login home, `~/.claude` exists or a global git identity is set. On a runner not yet isolated it fails the job with the command above — a red job, not a warning, because pull-request and release jobs share the runner (#3656).

.env is rewritten only when a managed line differs (other lines are kept), and a runner restarts to read it only when idle. tests/test-fleet-runner.sh proves the requirement’s test: a file one job writes — in the checkout, node_modules, beside the checkout, in RUNNER_TEMP, in HOME — is absent when the next job starts.

What isolation is not. It keeps one job’s leftovers away from the next. It is not what keeps a job out of your files: that is the runner’s account. A job can still read and write anything its account can, by absolute path. What that covers is in Trust boundary.

Disk and log retention (#1732)

A hosted runner vanishes with its disk; a fleet runner keeps _work (tool caches, action downloads, whatever a job left beside its checkout) and _diag (the runner’s and each job’s logs) until something deletes them. How long they may stay is declared next to the capacity, on the machine’s ci_runner:

"retention": { "work_days": 7, "diag_days": 14, "schedule": "daily 03:30 America/Los_Angeles" }

runner.sh hygiene --apply (and install) turns it into two things on the machine, both owned by bob-reap:

a reap.json in each runner home: _work/ older than work_days and _diag/ older than diag_days, owner ci-runner;
one systemd timer, bob-runner-reap, that runs reap.js <runner home> --apply --owner ci-runner for every runner home on the declared schedule. It is a system unit with User= set to the runner’s account, because the reaper deletes inside runner homes and those belong to that account.

The floor: bob-reap skips a runner home entirely while a job runs in it (a Runner.Worker from its bin/), and treats “cannot tell” as running. Only files are aged out, by mtime; the runner install (bin/, externals/, .runner, .credentials) is never a surface. Preview or run it on demand:

sudo systemctl start bob-runner-reap.service      # reap now, as the runner's account
journalctl -u bob-runner-reap.service -n 20       # what it deleted or skipped, and why
systemctl list-timers bob-runner-reap.timer       # when it runs next

The runner’s account cannot read your ~/.claude or your node. So hygiene keeps a copy of the reaper in .bob-reap in the account’s home, and the unit uses the node a runner ships with (externals/node*). The copy is refreshed whenever hygiene --apply finds the source changed; after a deploy that changes scripts/reap, re-run runner.sh hygiene --machine <id> --apply.

On a machine that declares no runner account, the timer is the login user’s own (systemctl --user list-timers bob-runner-reap.timer) and names the node found on PATH when hygiene ran. hygiene retires it when the machine declares an account.

Service control on the target

Each runner runs as a service (systemd on Linux, launchd on macOS) out of its runner home. Drive it through runner.sh, which resolves the home per repo:

scripts/fleet/runner.sh service status  --machine farm-01 [--repo <owner/name>]
scripts/fleet/runner.sh service stop    --machine farm-01
scripts/fleet/runner.sh service start   --machine farm-01
scripts/fleet/runner.sh service restart --machine farm-01

Each is idempotent — starting a running service or stopping a stopped one is a no-op — and --print shows the command without running it. By hand, the same is sudo sh -c 'cd ~bob-ci/actions-runner-bob && ./svc.sh status' (with the machine’s declared account, and start or stop in place of status).

Updates. GitHub self-updates the runner application, so there is no routine upgrade step. To move to a specific release — a pinned version, or a failed self-update:

RUNNER_VERSION=2.328.0 scripts/fleet/runner.sh update --machine farm-01   # omit RUNNER_VERSION for the latest release

It is a no-op when the runner already reports that version, refuses while a job is running (exit 3; --force overrides), and otherwise stops the service, unpacks the release over the install and starts it again. The registration (.runner, .credentials) and _work are not in the tarball, so nothing re-registers. A corrupted install is still repaired by a re-register: runner.sh install --machine farm-01 re-downloads and replaces in place.

When the runner is offline

A job assigned to an offline runner queues — it does not fail. There is no red check to notice and no error to read: the run simply sits in Queued. GitHub’s documented limit is that “a job can be in the queue for 24 hours before it is automatically cancelled,” so a weekend outage silently eats a day of delivery before anything visibly goes wrong. BoB makes this state loud (#1731): a preflight that asks will the next job run?, an automatic fallback to GitHub-hosted, a queued-job alarm that tells offline from busy, and an automatic return to the fleet runner once it is back.

Detect it

node ~/.claude/scripts/fleet/ci-locus.js preflight --project <name>   # will the NEXT job run? exit 0 yes · 1 would-change · 3 blocked · 4 not measured
node ~/.claude/scripts/fleet/ci-locus.js queue --project <name>       # is a job already parked? exit 3 = alarm, with its kind
scripts/fleet/runner.sh status                                        # exit 3 when CI targets a label with no online runner
gh run list --limit 5                                                 # a run stuck in `queued` corroborates

The preflight is what warp-drive runs itself on entry to its merging phase, so a run that hits an offline runner stops with a named cause instead of waiting out a merge-on-green timeout; bob warp status shows the verdict on its CI runner: line, and the fleet view (bob status --fleet / audit.js view) shows a QUEUED column in its CI locus table.

Read the alarm

Every alarm names its kind — that is how you tell the two “queued” states apart:

You see	It means	Do
`QUEUE ALARM (offline)` — `NO online runner carries that label`	Queued because offline. Nothing will ever pick the job up.	Recover below.
`busy` — `every online runner with that label is busy`	Queued because busy. The runner is up and working; the queue drains as jobs finish. Not an alarm.	Wait, or add a runner (`runner.sh install --machine <id>`).
`QUEUE ALARM (idle-runner)` — a runner is idle yet the job waits	The runner is up but not taking the job: the job’s `runs-on` labels do not match the runner’s, a runner-group restriction, or a stuck service.	Compare `gh run view <id> --json jobs --jq '.jobs[] \| {name, labels}'` with `runner.sh status`; `sudo ./svc.sh restart` on the machine.
`QUEUE ALARM (hosted)` — parked on `ubuntu-latest`	GitHub-side capacity. A billing outage does not look like this — it shows as a job that failed with zero steps (todo #1648).	Check githubstatus.com and the account billing page.
`NO VIABLE RUNNER`	The fleet runner is offline and hosted is dead (the newest completed run’s job never started). One report, both failures named.	Recover the runner below, or restore hosted Actions (todo #1648). Nothing was written.

The threshold is declared per project as _meta.ci.queue_alarm_minutes (default 10) in its provision manifest.

Recover it — restore delivery first, diagnose second

Let the preflight route around it: node ~/.claude/scripts/fleet/ci-locus.js preflight --project <name> --apply. When hosted is viable it clears CI_RUNNER_LABEL through runner.sh target hosted (verdict FALLBACK) so the next job runs on ubuntu-latest — within one job, never after a queue timeout. Warp-drive does this itself at Level 3; at Level 2 it prints the command for you.
Re-run the parked workflow — gh run rerun <id> (the alarm’s fix line names the oldest one); it picks up the hosted runner.
Now diagnose the machine: is it up, is it on the tailnet, is the service running (runner.sh service status --machine <id>)?
When it is healthy again, nothing to flip: the next preflight sees the declared farm locus, the online runner and the unset variable, and restores CI_RUNNER_LABEL (verdict RESTORED). The whole outage is exactly two GitHub writes. Until then the repo shows [drift] in bob provision status — that visibility is deliberate.

Manual override

Do not fall back automatically for a project: declare _meta.ci.fallback: "none" next to its locus. The preflight then reports DEGRADED (exit 3) with the command to run by hand instead of writing anything.
One run without the fallback: preflight --no-fallback or BOB_CI_FALLBACK=0.
Flip by hand: scripts/fleet/runner.sh target hosted / runner.sh target self-hosted-bob — still the only writer of the variable. A hand flip to hosted while the runner is online shows as drift and is reverted by the next reconcile --apply, not by the preflight (which only restores after a fallback).
Turn the guard off in warp-drive: _workflow.ci_runner_preflight: false (the merge-on-green no-checks / timeout verdicts remain the backstop).

If Actions is down or capped entirely — a different failure from this one — the fallback is Local Delivery Fallback.

Security constraints

A self-hosted runner is a machine on your network executing code from a repo. Four constraints keep that acceptable; the first is absolute.

The repo must stay private. On a public repo, anyone who opens a fork pull request can execute arbitrary code on the runner — GitHub warns against this explicitly, and it is a full compromise of the host, not a sandbox escape. runner.sh install refuses to register against a public repo, runner.sh target <label> refuses to point one at a runner label (no override — --force vouches for a runner that is coming up, never for the repo’s visibility; target hosted stays allowed as the way out), the preflight never restores a farm label on one (#3657), and runner.sh status prints a security warning if it ever finds runners attached to one. If this repo is ever made public, remove the runner before flipping visibility.
The runner has its own account. config.sh runs as the account the machine declares (ci_runner.user), and the service is installed for that account — never root, and never the account you log in with. A job runs with everything its account has, so that account holds nothing but runner homes. runner.sh install refuses any other arrangement and runner.sh status flags a runner that predates the rule (The runner’s account).
The registration token never enters an argument vector. It is streamed to the target over ssh stdin, and config.sh reads it from its environment (ACTIONS_RUNNER_INPUT_TOKEN), not from a --token argument, so it does not appear in ps output, a shell history, or a process listing on either machine. The removal token is passed the same way. Registration tokens are short-lived, but a leaked one is enough to attach a hostile runner to the repo. The privileged phase is separated precisely so no secret ever needs a sudo tty.
Secrets reach the runner only as GitHub-injected job environment. A self-hosted runner persists between jobs, so anything a job writes to disk is visible to the next one — which is why a workflow reads a credential only as ${{ secrets.NAME }} (GitHub injects it into that job’s environment, masked in logs, and it is gone when the job ends) and never from the host (~/.claude, .mcp.json, .dev.vars, a shell profile). What the three shipped workflows name, and where each is declared (#1729 AC-06, per the runner-hygiene rule in #1732):
- docs-deploy.yml → CLOUDFLARE_API_TOKEN / CLOUDFLARE_ACCOUNT_ID: inventory cloudflare-api-token-ci / cloudflare-account-id, home gh-secret paulirv/bigbrain:…, rotation Cloudflare API Token Rotation (P1Y). cloudflare/wrangler-action passes the token to wrangler as environment; wrangler persists nothing for a token (its ~/.config/.wrangler/ on farm-01 holds logs and caches, no config/ — checked 2026-09-16). BOB_CHECKOUT_TOKEN (bob-checkout-token) is optional on this repo.
- auto-release.yml → GITHUB_TOKEN (github-actions-token, automatic, per-run) and the BRIDGE_* pair (rdb-bridge-secret / rdb-bridge-worker-url, declared, not set as of 2026-09-15).
- ci.yml → none. Every entry lives in secrets-inventory.json with the workflow under consumers.workflows, so the declared home and the rotation path are findable from the workflow file. A new credential a workflow needs on the farm gets the same treatment before the workflow merges: an inventory entry (home, consumers.workflows, rotation) and a secrets.* reference — never a file on the runner.
Nothing a job wrote outlives the retention. The runner masks every secrets.* value in its logs, but a job can still print or write something sensitive. The job-completed hook empties the checkout, RUNNER_TEMP and HOME when the job ends (Workspace isolation), and _diag logs age out after diag_days (Disk and log retention) — so run logs are retention-bounded, not kept forever.

The runner is also not a VM per job: the filesystem persists across jobs. Workspace isolation empties what a job wrote before the next one starts, which is what makes a green farm run mean what a green hosted run meant; keep the hosted path as the reproducible reference all the same.

Trust boundary: what a CI job on the farm can reach

Moving CI onto farm-01 moved it inside the fleet’s trust boundary. A hosted job ran in a throwaway VM with no route to anything of ours; a farm job runs on the farm, on the tailnet, as the runner’s account. The account decides what a job can read and write outside its workspace. runner.sh status says which account each runner runs as.

A runner on its own account (The runner’s account):

A farm CI job…	Reach	Why
reads your home: `~/.claude`, the GitHub login, ssh keys, the bridge secret, project checkouts	no	another account; your home is not readable by others (mode 750 on farm-01 — check with `stat -c %a ~`)
writes your hooks, `authorized_keys`, shell profile or `settings.json`	no	another account, with no sudo
reads what any local account can: `/etc`, `/usr`, `/proc`, world-readable files	yes	an ordinary Unix account
writes anywhere in the runner account’s home: its own runner install (`bin/`, `.env`, `bob-job-hook.sh`), the reaper copy, another repo’s runner home	yes	every runner on the machine runs as the one account
writes `/tmp`, `/var/tmp`, `/dev/shm`	yes	world-writable
leaves files for the next job	no in its workspace, temp dir and HOME; yes everywhere else it can write	the job hook empties those three and nothing else; a file written to the two rows above stays until someone deletes it
leaves a process running after the job	yes, as the runner’s account	the runner ends the processes it can trace to the job; one that detaches and drops `RUNNER_TRACKING_ID` keeps running
reaches tailnet machines (ssh, the dashboard, other services)	yes	the runner has the host’s network, tailnet included
sees a `secrets.*` value	only the ones its own workflow names, for the life of the job	GitHub injects them, masked
uses more than the reserve’s CPU and memory	no	`bob-ci.slice` (Capacity)

So a job that turns hostile can make itself persistent inside the runner account: it can change the job hook or the runner install, and the next job on that machine runs with the change. It cannot read or write your files. Removing the runners, deleting the account’s home and running install again resets it.

A runner on the account you log in with (status flags it): every “no” in the first two rows is a “yes”. The job reads your GitHub login, ssh keys and bridge secret by absolute path, can write a hook or an authorized key that outlives it, and can sudo if you can. The isolated HOME and the ssh setting only stop a job reaching those by accident. Move the runner: Move a runner off the account you log in with.

What that means for a dependency change. Code a CI job runs — a new npm package, a post-install script, a GitHub Action, a test fixture — runs with the reach in the “yes” rows. So on a repo whose CI runs on the farm:

review a dependency or action addition as code that will run on farm-01, not as code that runs in GitHub’s sandbox; pin actions to a release or SHA;
keep the repo private — the one absolute rule (Security constraints): a fork PR on a public repo is anyone’s code on this machine;
when in doubt, run that change’s CI on hosted (runner.sh target hosted --repo <owner/name>), then flip back once it is trusted.

Anything that has to be kept from CI entirely belongs on a machine without a runner.