Client Farms

Client Farms

farm-01, for someone else, in their account. A client farm is a BoB farm that a client owns: its server, its cloud bill, its tailnet, its GitHub org, its Claude seats. It runs the same night shift as farm-01 — see The Farm — and is built from a declaration in the client’s own operator repo, so it can be rebuilt from nothing without IWPI at the keyboard. This page explains the design. The work that builds it is the client-farm capability, #2216.

What a client farm is

A client farm is one Linux machine in a cloud account the client pays for. On it run:

• BoB, at a pinned stable release, with its own BOB_HOME and its own GitHub login. It runs the tick (bob tick), so the client’s delegated work gets warp-drive runs at night, exactly as on farm-01.
• A CI runner, registered to the client’s GitHub org with the client’s credential, so the client’s repositories run their checks on the farm rather than on IWPI’s runner or on hosted Actions minutes.
• The host provider’s workloads: the hardened OS, the toolchain, Tailscale, the systemd timers and watchdogs, and the bob workload that installs and checks BOB_HOME.

The farm reports the same two records as farm-01: host-status/1 from the host provider and a fleet snapshot from BoB. So supporting a client farm uses the contracts that already support farm-01, and never a login to it.

Nothing on a client farm comes from IWPI’s data. The client’s identity, machines, projects, workloads and secret names come from the client’s operator repo (Operators); the secret values come from the client’s own secret store (#2215).

The ownership seam

A client farm has three owners. Each owns one layer, declares it in one place, and touches the others only through a contract.

Layer	Owner	Declared in	Touches the next layer by
The OS and the workloads — packages, hardening, Tailscale, the runner, timers, the bob workload	The host provider (thefarm’s tooling, at a release tag)	its own repo, as tooling only	running BoB’s deploy.sh / bob update, and reporting host-status/1
BOB_HOME — skills, scripts, hooks, registry, settings hook wiring	BoB (this repo, at a pinned release)	deploy.sh, bob update	reading the operator repo through scripts/lib/operator-resolve.js
The declaration — who the client is, which machine is the farm, its size and region, its workloads, projects, people, routes, secret names, the BoB and tooling versions to run	The client’s operator repo (#2267)	operator.json, machines.json, provisions/, and the farm declaration (#2278)	being read; it runs nothing

The first two rows are the seam BoB Identity already describes for farm-01. The client farm adds the third owner explicitly: on farm-01 the declaration is still partly inside thefarm, and on a client farm it never is.

Tooling apart from data

Today thefarm mixes two things. Its scripts, playbooks, unit templates, runbooks and farm CLI are tooling any operator could run. Its workload registry (infra/config/workloads.json), project list, inventory vars, sandbox config, egress allow-list, SOPS recipients and encrypted secrets are IWPI’s farm data. Its bootstrap defaults to cloning IWPI’s repo. A client following that bootstrap would receive IWPI’s workloads and IWPI’s encrypted secrets — the cross-tenant leak #2215 exists to prevent.

Decision #4219 splits them the way decision #2266 split BoB:

• The tooling is a product. It holds no host name, workload, project, recipient or secret, and is released by tag.
• The farm data is operator data. It lives in the client’s operator repo, next to the rest of the client’s declaration, with the SOPS creation rules holding only public age recipients.
• Secret values live in the client’s own store. The operator repo names each secret and its source (operator.json secrets[]), never its value, and bob secrets check proves each one is present without reading it (#2286).

A client gets the tooling the way it gets BoB: its operator repo names the upstream and the release tag to run, and the bootstrap fetches that tag. The bootstrap’s one input is the client’s operator repo. There is no default that points at IWPI, and with no operator repo the bootstrap refuses. IWPI is the first operator under the same rule: its farm data moves into its own operator repo. The thefarm side of the change is paulirv/thefarm#157.

How a client farm is built

The build is one run, in this order. Every input comes from the client’s operator repo or the client’s secret store.

1. Order the server. The cloud-provider layer creates a VM of the declared size, in the declared region, from the declared Ubuntu LTS image, with the client’s SSH key, and returns its address. This is the only step that knows which cloud the client uses.
2. Harden and equip it. The host-provider tooling, at its declared tag, locks down SSH, the firewall and unattended upgrades, then installs the toolchain (Node.js, Docker, Git, tmux).
3. Join the client’s tailnet with the client’s auth key. From here the farm is reached over Tailscale only.
4. Install BoB. The bob workload clones BoB at the pinned release, writes the .bob-operator pointer to the client’s operator repo, runs deploy.sh, and checks the result with bob doctor (#2279).
5. Register the runner to the client’s GitHub org with the client’s registration credential (#2280).
6. Start reporting: host-status/1 and the fleet snapshot, to the destination the client declares (#2281).

The build is done when bob doctor is green and bob preflight passes on the new machine.

The cloud-provider layer

Only step 1 knows which cloud the farm lives in. Everything from step 2 on talks to an Ubuntu machine over SSH and then Tailscale, and does not care whose data centre it is in. So the provider-specific surface is one thin layer in the host-provider tooling, and the client’s choices (size, region, image) are data in its operator repo, not code.

Hetzner is the first provider, because farm-01 already runs there. Its layer is what thefarm’s bootstrap runbook does by hand today:

Surface	Hetzner today (farm-01)
The account	a Hetzner Cloud project and an API token, used through the hcloud CLI
The VM order	hcloud server create with a type (cx32: 4 vCPU, 8 GB), a location (ash) and a name
The image	ubuntu-24.04
Access at birth	an SSH public key registered with Hetzner and injected at create time
The network	a public IPv4 address, read back with hcloud server ip; the host firewall and the tailnet are set up by step 2 and 3, not by the provider
Rebuild	delete the server and create it again with the same inputs

What a second provider has to supply. Any cloud qualifies that can do these, from a CLI or an API, with the client’s own credential:

1. Create a VM of a declared size in a declared region, by name.
2. Boot it from a current Ubuntu LTS image.
3. Inject an SSH public key so the first login works without a password.
4. Return the address the bootstrap reaches it on.
5. Delete it, so a rebuild can start from nothing.
6. Optionally, a provider firewall that admits only SSH until the machine has joined the tailnet. It is a second wall, never the only one: the host firewall from step 2 is.

That is the whole contract. A provider needs no special network, volume, load balancer or managed service, because the farm keeps no state on the machine that is not declared elsewhere.

What the client supplies

The only manual inputs to a client farm are the client’s own credentials and accounts. Everything else is either declared in the client’s operator repo or done by the build. Each input belongs to a capability, and that capability owns how it is declared and checked:

Input	Used for	Owned by
Cloud account — an account with the chosen provider and an API credential	step 1, ordering and rebuilding the server	#2215 (the credential stays in the client’s store); the provider choice itself is the farm declaration, #2216
Tailnet — the client’s Tailscale network and an auth key for it	step 3, joining the farm to the client’s network	#2215 (the auth key is a client secret)
GitHub org — the org that owns the client’s repos, and the farm’s login and runner credential for it	the farm’s forge login, and step 5, registering the runner	#2214 (the org is the operator’s forge owner) and #2215 (the token and runner credential are client secrets)
Claude seats — the Claude Code accounts the farm signs in with	every warp-drive run the tick starts	#2215 (the login is a client credential)

Each one is declared by name and source in the client’s operator.json secrets[], and bob secrets check reports any that are missing, with where it should come from, before the build needs it. None of them ever passes through an IWPI machine or repository.

How it is updated

Each owner updates its own layer, and only through its own command:

• BoB moves only by bob update to a stable release. A client farm never runs a source checkout or an untested nightly; the release it runs is the one its operator repo pins (#2279).
• The host-provider tooling moves when the operator repo’s pinned tag moves, and is applied by the tooling’s own idempotent bootstrap.
• The OS is patched by the host provider’s patch-and-reboot runbook.
• The declaration changes only by commits to the client’s operator repo, reviewed there.

How it is rebuilt

Losing the server is an inconvenience, not an outage. Because the server holds no state that is not declared somewhere else — the operator repo holds the declaration, the forge holds the code and the issues, the secret store holds the values — a rebuild is the build again, against an empty account. The host provider’s rebuild runbook runs the same steps in the same order, and the farm counts as rebuilt when it passes the same checks as a new one. The rebuild acceptance requirement (#2282) sets the time budget it has to meet and proves it on an empty client account.

Related

• The Farm — what the farm does each night, on farm-01 or a client’s.
• Operators — the operator repo a client farm is declared in.
• BoB Identity → Ownership seam — the BoB / Fleet / thefarm split this page extends.
• Decision #4219 (tooling apart from data) and decision #2266 (operator data apart from BoB), which it mirrors.