Fleet Snapshot Timer

Fleet Snapshot Timer

audit.js view and bob viz read the snapshots each machine publishes to $BOB_FLEET_SNAPSHOTS (default $BOB_HOME/fleet/snapshots). A snapshot is only as fresh as the last audit.js snapshot on that machine, so every fleet machine runs it on a timer (#2067).

Commands

audit.js snapshot --install-timer [--interval <min>] [--dry-run] [--json]
audit.js snapshot --uninstall-timer [--dry-run] [--json]

audit.js here is scripts/fleet/audit.js, or the deployed copy at $BOB_HOME/scripts/fleet/audit.js. Both flags hand off to the engine scripts/fleet/snapshot-timer.js, which also has status and render:

node scripts/fleet/snapshot-timer.js install   [--interval <min>] [--platform systemd|launchd] [--dry-run] [--json]
node scripts/fleet/snapshot-timer.js uninstall [--platform systemd|launchd] [--dry-run] [--json]
node scripts/fleet/snapshot-timer.js status    [--platform systemd|launchd] [--json]
node scripts/fleet/snapshot-timer.js render    [--interval <min>] [--platform systemd|launchd] [--out <dir>]

Flag	Meaning
--interval <min>	Minutes between snapshots. A positive integer. Default 60.
--platform systemd|launchd	Pick the backend. Default: launchd on macOS, systemd on Linux. Any other OS exits 2.
--dry-run	Render the units and report what would be written and run. Writes nothing and runs nothing.
--json	Print one snapshot-timer/1 document on stdout.
--out <dir>	render only: write each unit into <dir> by file name instead of printing it.

Install is idempotent. A unit already on disk with the same content is reported unchanged and not rewritten. The timer is enabled again either way, so running it twice converges. Uninstall stops the timer and removes the unit files. When nothing is installed, it reports every unit as absent and exits 0.

What gets installed

Both backends run the deployed $BOB_HOME/scripts/fleet/audit.js, the copy that exists at boot wherever the source checkout lives. They pass HOME, BOB_HOME, BOB_FLEET_SNAPSHOTS and a PATH that starts with the directory of the node that ran the install.

Platform	Unit files	Enabled with
Linux (systemd)	~/.config/systemd/user/bob-snapshot.service (oneshot) and bob-snapshot.timer (OnBootSec=2min, OnUnitActiveSec=<interval>min)	systemctl --user daemon-reload, then systemctl --user enable --now bob-snapshot.timer
macOS (launchd)	~/Library/LaunchAgents/com.bob.fleet-snapshot.plist (StartInterval, RunAtLoad; log at $BOB_HOME/fleet/snapshot-timer.log)	launchctl bootout (a missing agent is fine), then launchctl bootstrap gui/<uid>

The templates live in templates/systemd/ and templates/launchd/. templates/systemd/render.js renders them: it fills {{NAME}} placeholders and refuses to leave one unfilled or to accept a value containing a newline. --xml escapes values for a plist. The farm tick units (#2056) use the same renderer, and so do the always-on dashboard units: the BoB source checkout’s (#3001), and one per project bob serve routes, rendered from the same two templates with the label and checkout filled in (#2407, see bob viz → Project units).

Lingering (Linux)

A systemd user timer runs only while the user’s manager is running. With lingering off, that stops when the last session logs out, so a headless machine stops publishing. The install report includes linger, read with loginctl show-user $USER -p Linger. When it is no, the report tells you to run loginctl enable-linger $USER. If that needs root, run it with sudo.

Verify an install

Don’t wait for the first interval. Trigger one run, then confirm that it published.

Linux (systemd):

systemctl --user start bob-snapshot.service                     # one run now
systemctl --user show bob-snapshot.service -p Result             # Result=success
systemctl --user list-timers bob-snapshot.timer --no-pager       # NEXT is about one interval away
journalctl --user -u bob-snapshot.service -n 4 --no-pager        # "snapshot written: …/fleet/snapshots/<id>.json"
loginctl show-user "$USER" -p Linger                             # Linger=yes, or the timer stops at logout

macOS (launchd):

launchctl kickstart "gui/$(id -u)/com.bob.fleet-snapshot"       # one run now
tail -n 3 ~/.claude/fleet/snapshot-timer.log                     # "snapshot written: …"

On either OS, node scripts/fleet/snapshot-timer.js status should report installed (every 60 min). audit.js view --no-live should then show the machine as cached, not never. A second --install-timer reports every unit as unchanged.

Enabling lingering (loginctl enable-linger $USER) usually works without sudo for your own user. If polkit refuses it, run it with sudo.

Freshness: audit.js view and bob doctor

audit.js view grades each machine by its newest snapshot:

Status	Meaning
live	Built on this machine, or generated on the remote machine over Tailscale, just now.
published	Read just now over Tailscale from the snapshot the remote machine last published to its own snapshot dir, because that copy is younger than the freshness window. The row’s age_seconds is that snapshot’s age.
cached	The newest published snapshot is younger than the freshness window.
cached(stale)	The newest snapshot is older than BOB_FLEET_STALE_SECONDS (default 86400, 24 h). --json rows carry stale: true.
never	There is no live result, and this machine has never published a snapshot here.

How a live pull reads a remote machine

For every machine but this one, view (and audit.js pull <id>) works in two steps over Tailscale SSH (#3609):

1. The published copy. It reads the snapshot the machine last published — $BOB_FLEET_SNAPSHOTS/<id>.json on that machine, else <its BOB_HOME>/fleet/snapshots/<id>.json — with a short timeout (BOB_FLEET_PUBLISHED_TIMEOUT_MS, default 15000). When it parses, its machine.id is that machine’s id, and it is younger than BOB_FLEET_STALE_SECONDS (and no more than 5 minutes in the future, so a clock running ahead cannot pin a copy as fresh), it is used and cached here, and the row’s source is published. A copy that fails any of these is ignored, and step 2 runs. A machine whose timer is installed is answered this way, without waiting on a full snapshot.
2. The generator. Otherwise it runs audit.js snapshot --stdout on the machine and waits up to BOB_FLEET_PULL_TIMEOUT_MS (default 180000, 3 min — a full snapshot on farm-01 takes longer than 30 s). When that times out, the error names the timeout and the variable, and the row falls back to the cached copy, or never.

audit.js pull <id> --cmd '<command>' skips both steps: it runs the one command and returns its output.

--json rows also carry reachable: false when a live pull was tried and failed, null when none was tried (--no-live). The summary line counts machines that never published, stale ones, and ones unreachable live. When any machine is never or stale, it names the fix: install the timer on that machine. bob status --fleet grades a never or stale machine as drift.

Each snapshot’s project rows also carry projects[].warp ({session, phase, started_at, tokens}) and projects[].last_commit ({sha, at, subject}). Both are read by scripts/fleet/project-live.js, the same reader the fleet feed uses for live rows (#2069). A snapshot published before #2069 has neither key, and the feed shows those values as not measured.

Each project row also carries projects[].activity (#2154): {dirty, commits_by_day}, where commits_by_day is 30 commit counts, oldest first. Element i counts the commits in the 24 hours that ended 29 - i days before the snapshot was read, so the last element is the last 24 hours. dirty is true when git status --porcelain lists anything. The reader is activityCondition in scripts/fleet/project-live.js: one git log --since=30.days --format=%ct per checkout, reusing the status read the row already makes for dirty (a count). It makes no network call. A repo with no commits in range, or none at all, has 30 zeros. A missing checkout or a directory outside git has activity: null, and so does a snapshot published before #2154. last_commit and branch are not repeated inside it. The bob viz Activity pane reads it through the feed.

Each present project row also carries rituals (#2232): {overdue, due, upcoming, unknown, overdue_keys, due_keys} from one ritual.js due --all read per snapshot, or null when the engine measured nothing (no ritual vocabulary in BOB_SOURCE, or a _meta.rituals that check-rituals refuses). Last runs are read from each checkout’s ritual-run issues. That is the one forge read a snapshot makes for rituals, and it is skipped when this machine’s forge signal is unreachable. Every dated ritual is then unknown, never overdue. $BOB_FLEET_RITUAL_RUNS names a ritual.js --runs file to use instead (offline use, tests). summary.rituals sums the counts and keeps them by_project. audit.js view renders a Rituals table: one row per machine plus a FLEET row that counts a project checked out on several machines once, taking the machine with the fewest unknown. It names each overdue ritual with the machine that reported it. A snapshot published before #2232 has no rituals key and adds no table row.

Each snapshot also carries last_tick: the newest bob tick digest on that machine, as {file, at, digest, markdown} (the tick-digest/1 document and its rendered .md), or null when no tick has written one there. A tick writes its digest only locally (#2054), so this is how a tick on farm-01 reaches the reports pane on another machine within one snapshot interval (#2073). A snapshot published before #2073 has no last_tick key, and the pane names that machine as not measured.

bob doctor prints the timer as one INFO line. It is never graded and never changes the exit code.

  INFO  snapshot timer: installed (systemd, every 60 min), timer active, linger no — fires only while logged in (loginctl enable-linger $USER) (#2067)
  INFO  snapshot timer: not installed (launchd) — audit.js snapshot --install-timer (#2067)

The same summary appears as snapshot_timer in bob doctor --json: {platform, installed, current, active, interval_minutes, linger, error}. current: false means the installed units no longer match what the current source renders. Re-run the install to fix it.

Backup freshness (backups, #3086)

Every snapshot carries, per present project, the freshness of the D1 backup its dev.json declares, as backups. It is read through bob backup status --json by scripts/backup/freshness.js, so the snapshot and the command never disagree:

Key	Meaning
projects[].backups	{declared, verdict, stale, newest_at, age_hours, size, due_at, reason, source, envs}, or null when the project declares no backup. verdict is fresh, stale or unknown. An unreadable bucket is unknown with stale: null — never fresh.
summary.backups	{declared, fresh, stale, unknown, stale_names, unknown_names, by_project} across this machine’s projects; null when none declares a backup.

audit.js view prints the roll-up as a Backups table, one row per machine: DECLARED FRESH STALE UNKNOWN PROJECTS, each stale project with its age. BOB_FLEET_BACKUP_PROBE=0 skips the bucket read and records a declared backup as unknown; BOB_BACKUP_STATUS_TIMEOUT_MS (default 60000) bounds each read, and a read that times out is unknown. The notice that a miss sends, and the field reference, are in Dev lifecycle.

Actions budget (summary.actions_budget, #1695)

Every snapshot carries the GitHub Actions minutes each distinct owner of this machine’s projects has used against the included quota, as summary.actions_budget: {verdict, warn_pct, owners[]}. Each owner record is {owner, kind, verdict, used_minutes, included_minutes, paid_minutes, pct, warn_pct, source, reason, fix, checked_at} and verdict is the worst of them. A verdict is ok, warn, over, unknown or unavailable. A token that cannot read billing is unavailable with its fix, never ok. The field is null when no project has a GitHub origin or when BOB_FLEET_ACTIONS_PROBE=0. Reads go through a 6-hour per-owner cache, so a snapshot costs at most two gh calls per owner a few times a day.

audit.js view prints the Actions budget table, one row per owner across the fleet: OWNER VERDICT USED% USED INCLUDED NOTE. The verdicts, the scope a token needs and the CLI are in Actions budget.

--json

{
  "schema": "snapshot-timer/1",
  "action": "install",
  "platform": "systemd",
  "dry_run": false,
  "interval_minutes": 60,
  "snapshots_dir": "/home/farm/.claude/fleet/snapshots",
  "units": [
    { "name": "bob-snapshot.service", "path": "/home/farm/.config/systemd/user/bob-snapshot.service", "state": "written" },
    { "name": "bob-snapshot.timer", "path": "/home/farm/.config/systemd/user/bob-snapshot.timer", "state": "unchanged" }
  ],
  "commands": [
    { "argv": "systemctl --user daemon-reload", "exit": 0, "ok": true, "tolerated": false, "stderr": null }
  ],
  "linger": "no",
  "installed": true,
  "active": "active",
  "ok": true,
  "error": null
}

Key	Meaning
action	install, uninstall or status.
units[].state	Install: written, unchanged, would-write (dry run). Uninstall: removed, absent, would-remove. Status: current, differs, absent.
commands[]	Every systemctl/launchctl call, in order, with its exit. Under --dry-run they are listed with exit: null and not run. A tolerated failure, such as bootout of an agent that is not loaded, does not fail the run.
linger	yes, no or null. null means not systemd, or loginctl did not answer.
installed	Every unit file is present.
active	systemd: the output of systemctl --user is-active bob-snapshot.timer. launchd: loaded or not-loaded. null under --dry-run.
error	null, or {code, message} when the run exits 2.

Exit codes

Code	Meaning
0	Installed, removed, rendered or reported. This includes “nothing to remove”.
1	A systemctl or launchctl step failed. The failure is listed under commands.
2	Usage error: bad --interval, unknown --platform, or no timer backend on this OS.

See also

• bob status: the fleet scope reads these snapshots through audit.js view
• Farm Tick: the tick also publishes a snapshot once per cycle
• tests/test-snapshot-timer.sh: renders both unit types from fixtures and diffs them against golden files