Verification System

Verification & Testing System

TL;DR. make check validates the system (deps, schemas, symlinks, hooks, provisions); make test runs the suites; make ci runs both — what GitHub Actions runs. One command to verify everything.

Bob’s infrastructure-as-code verification layer. One command to validate the entire system.

make check    # verify everything
make test     # run all tests
make ci       # both (what GitHub Actions runs)

Quick Start

cd ~/projects/bigbrain    # or wherever BOB_SOURCE is
make help                 # see all targets
make check-deps           # am I set up correctly?
make check                # is everything healthy?
make test                 # do all tests pass?

Targets

Verification (make check)

make check runs all five checks in order. Stops at first failure.

Target	What It Checks	Speed
check-deps	node >= 18, jq, git, make installed	instant
check-schemas	All JSON configs valid against schemas	instant
check-manifest-meta	Every provisions/<project>.json carries its own _meta.project — never the _default template’s (#1869)	instant
check-symlinks	Symlinks in the current repo resolve (--all / make check-fleet for every project)	~1s
check-hooks	All hooks in settings.json exist + executable	instant
check-provisions	The current repo’s symlinks match its manifest (--all for the fleet)	~1s
check-scenarios-drift	Every command/flag/path/count named by the workday scenarios page still ships (#843)	instant
check-abs-paths	No machine-absolute path (/Users/<name>/, /home/<name>/) in any synced or ledgered file, and no absolute-target symlink in a synced tree (#1924). Allow-list with a why per entry; --apply migrates	instant
check-host	This machine’s BOB_HOME matches source on every deploy-managed surface, every symlink resolves here, toolchain preconditions hold, settings hooks converged (#1921). Skipped under $CI — it grades the host, not the checkout. Alias make doctor / bob doctor	~0.2s
check-fleet	check-symlinks + check-provisions across all registered projects (#294)	~2s

Testing (make test)

Target	What It Tests	Count
test-schemas	Schema validator unit tests (type, enum, pattern, etc.)	34
test-warp-drive	Warp-drive state machine transitions	99
test-checks	Integration tests for all check scripts	13
test-scenarios-drift	The scenarios drift-check catches a broken reference and honours the allowlist (#843)	5
test-render-scenarios	The scenarios two-render split strips/keeps the friction margin and is idempotent (#844)	5

CI (make ci)

Runs check then test. This is the target GitHub Actions calls on every push and PR to master.

---

Check Details

check-deps

Verifies the four required tools are installed:

  PASS  node 25.5.0 (>= 18)
  PASS  jq 1.7.1
  PASS  git 2.48.1
  PASS  make (GNU Make 3.81)

If something fails, the output includes a fix command:

  FAIL  node 16.20.0 (need >= 18)
        Fix: Install Node.js 18+ via nvm or brew

check-schemas

Validates every JSON configuration file against its JSON Schema definition:

File	Schema
projects.json	schemas/projects.schema.json
settings.json	schemas/settings.schema.json
provisions/*.json (17 files)	schemas/manifest.schema.json

If something fails, the output shows the exact path and error:

  FAIL  provisions/bodmail.json
        $._meta.project: expected type string, got number
        $.skills[3]: duplicate item "webapp-testing"

check-manifest-meta

Guards manifest identity (#1869). cdprov --init --auto once copied provisions/_default.json verbatim when no stack was detected, leaving _meta.project: "_default" and _meta.path: null — a name that collides across projects in audit.js / cdprov status and a path nothing can resolve. The generator now stamps both; this gate keeps the corpus honest:

• FAIL — _meta.project missing, "_default", or not equal to the manifest’s basename.
• WARN (non-fatal) — _meta.path is null; set it ("$HOME/projects/<name>" convention) or re-run cdprov --init --auto from the project.

Exempt: _default*.json (templates), _bob-home*.json (the BOB_HOME surface, #411) and ports.json (the port-slot registry, #197).

To fix: Edit the JSON file to match the schema. The error path ($._meta.project) tells you exactly where the problem is.

check-symlinks

Scans every project listed in projects.json, finds all symlinks in each project’s .claude/ directory, and verifies each one resolves to a real file.

  FAIL  /Users/you/Sites/myproject/.claude/commands/sprint.md -> /Users/you/.claude/registry/commands/sprint.md (dangling)

To fix dangling symlinks:

# Option 1: Re-provision the project (recommended)
cdprov --refresh /path/to/project

# Option 2: Remove the stale symlink manually
rm /path/to/.claude/commands/sprint.md

Projects that don’t exist on disk or lack a .claude/ directory are silently skipped.

check-hooks

Reads settings.json, extracts every hook command path (from SessionStart, PreToolUse, PostToolUse, Stop), and verifies:

1. The script file exists
2. The script is executable (chmod +x)

Also checks the statusLine.command if configured.

  PASS  /Users/you/.claude/hooks/temporal-context.sh
  FAIL  /Users/you/.claude/hooks/missing-hook.sh
        Fix: File not found — remove from settings.json or create the script

To fix:

# If the hook should exist:
chmod +x ~/.claude/hooks/the-hook.sh

# If the hook was removed:
# Edit settings.json and remove the hook entry

check-provisions

For each project in projects.json, loads its manifest (provisions/<project>.json or provisions/_default.json) and verifies that every declared skill, command, and agent is:

1. Present in the registry (registry/skills/, registry/commands/, registry/agents/)
2. Symlinked into the project’s .claude/ directory
3. Not dangling

  PASS  bodmail (bodmail.json)
  FAIL  myproject: commands/sprint declared in _default.json but not symlinked
  FAIL  myproject: agents/code-expert symlink is dangling

Failure types and fixes:

Failure	Meaning	Fix
declared but missing from registry	Manifest references something that doesn’t exist	Remove from manifest or add to registry
declared but not symlinked	Item exists in registry but wasn’t linked to project	Run cdprov --refresh /path/to/project
symlink is dangling	Symlink exists but target was moved/deleted	cdprov --refresh if the target still exists; cdprov --prune if it’s gone/cross-machine (see below)

check-abs-paths

State that deploy.sh syncs to every machine, or that a ledger records, rots the moment it carries one machine’s home directory. The two recorded shapes are a symlink inside a synced tree whose target is absolute — rsync -l copies the link verbatim, so a runbook link written on the Mac pointed at /Users/paulirving/... on the farm (#1914) — and a manifest whose _meta.path exists on one machine only (#1919). The #1475 LINK_ROOT rule (see the handbook) removes the cause for registry links at write time; cdi’s universal links and cdprov’s universal retargets follow it too (every link BoB writes targets $BOB_HOME, never the source checkout), and check-abs-paths catches everything else at commit time.

check-abs-paths runs scripts/checks/abs-path-lint.js:

• Scope — skills/, registry/, runbooks/, templates/, provisions/, scripts/, hooks/, bin/ and every *.json at the repo root. docs/ is deliberately excluded: prose quotes real paths as examples.
• Files fail on any literal /Users/<name>/ or /home/<name>/. Symlinks fail on any absolute target — a synced link cannot be derived from $HOME on the receiving machine, so it must be relative.
• Allow-list — scripts/checks/abs-path-allow.json. Each entry is { "path", "pattern"?, "why" }; path is a file or directory prefix, pattern an optional substring the offending line must contain. An entry without a why is a config error (exit 2) — this is a list of documentation examples, not a suppression list, and there is no implicit comment exemption (a comment is where a copy-pasted real path hides best).
• Migration — --apply (dry-run by default; the report always shows what it would do) rewrites a JSON string value that is an absolute home path to the $HOME/... form the hand-authored manifests use and provision.sh manifest_project_path writes (#1869), and rewrites a symlink whose absolute target lies inside the repo to the equivalent relative target. Each rewrite is reported; a hit in a script, or a link pointing outside the repo, is listed as unmigratable for a human. Idempotent.
• Contract — --json emits abs-path-lint/1: {hits[], allowed, rewrites[], unmigratable[], ok}. --root DIR lints another tree (fixtures). Exit 0 clean · 1 hit(s) remain · 2 usage/config error.

abs-path-lint — /Users/.../bigbrain
  scope: skills registry runbooks templates provisions scripts hooks bin *.json   allow-list: scripts/checks/abs-path-allow.json (1 allowed hit(s))
  would rewrite 1 ledgered path(s) (dry run — pass --apply):
    [would-rewrite] provisions/seebod.json _meta.path: /home/farm/Sites/seebod -> $HOME/Sites/seebod
  FAIL  1 machine-absolute path(s) in synced/ledgered files:
    provisions/seebod.json:4  /home/farm/  "path": "/home/farm/Sites/seebod",

Where a path is legitimately machine-specific, derive it at use time — $BOB_HOME for deployed tooling, $HOME for project roots (the port ledger’s contractHome, projects.json), $BOB_SOURCE for the checkout — or record a project id and resolve it through projects.json on the current machine.

check-host (bob doctor)

make check grades the checkout; nothing graded the deployed runtime a machine actually runs (#1914). A laptop can sit months behind source on scripts/, carry registry symlinks that point at another user’s home, or run the real gh ahead of the forge shim — and every other check stays green. check-host runs scripts/checks/host-drift.js against this machine’s BOB_HOME (#1921). Same entry point from bin/: bob doctor [--json] [--project <dir>] [--section a,b].

Four sections, each graded independently and each naming its own remediation:

Section	What it grades	Fix it names
surfaces	Every deploy-managed directory and file — the list is derived from deploy.sh’s own sync_dir / sync_file lines, so the check and the deploy cannot disagree about what is managed. Reports missing / stale / extra (extras count: deploy.sh syncs with --delete). Exclusions are an explicit list (.DS_Store, __pycache__, node_modules, .git), never a heuristic	scripts/deploy.sh
symlinks	Every symlink under BOB_HOME (runtime dirs like projects/, cache/ skipped) and under the current project’s .claude/ resolves on this machine. A dangling target, or a target under a foreign home (/Users/someone-else/…, /home/… on a Mac), fails and names the link	scripts/deploy.sh (BOB_HOME) · cdprov refresh (project links)
toolchain	node, jq, gh on PATH; BOB_HOME/bin/gh resolves ahead of the real gh (the forge shim must win, #1748); the one guarded bob-shell.zsh line is in ~/.zshrc	install the tool · scripts/bob-install.sh
settings_hooks	settings.json hook convergence — hook-drift.js (#1442/#1918) included as one section, not re-implemented	scripts/deploy.sh (reconciles .hooks)

Exit code is the answer: 0 converged, 1 drift, 2 error/usage. --json emits a versioned bob-host-drift/1 document (verdict, drift_count, sections[]), which the fleet snapshot carries per machine as summary.host_drift and audit.js view rolls up as a “BOB_HOME drift” table — so a machine behind source is visible from any other machine, with the fix named.

BOB_HOME drift — /Users/you/.claude vs /Users/you/projects/bigbrain
  FAIL  surfaces: 2/23 surface(s) behind source (4 file delta)
        scripts/                 missing 1  stale 2  extra 0
        CLAUDE.md                missing 0  stale 1  extra 0
          missing  scripts/fleet/settings-hooks-reconcile.js
          stale    scripts/deploy.sh — content differs from source
        Fix: scripts/deploy.sh
  PASS  symlinks: 134 link(s) resolve on this machine
  PASS  toolchain: node, jq, gh present; forge shim first on PATH; shell guard installed
  PASS  settings_hooks: 25 declared hook(s) wired

Drift: 4 — run scripts/deploy.sh, then re-run make check-host.

Not graded in CI ($CI set → SKIP), for the same reason as check-hooks scope 3 (#1708): the verdict would depend on when the runner host last ran deploy.sh, not on the commit under test. The signal is not lost — the fleet audit owns per-machine drift.

check-scenarios-drift

Keeps the narrative workday scenarios page honest: it re-grounds every BoB surface the page names against the actually-shipped inventory, so the page can’t quietly rot into fiction against BoB’s daily churn (#843). The shipped command/skill/agent inventory is reused from the same harvester the docs-site reference generator uses (scripts/docs-site/lib/gen-reference.js) — the check and the reference can never disagree about what ships. It is deterministic and side-effect-free (pure filesystem reads, no network, no deploy), consistent with the “audits stay cheap” rule.

What it verifies — three backticked syntactic classes plus prose counts:

Class	Example	Grounded against
Slash command	`/warp-drive`	a shipped command or skill name
Repo file/dir path	`scripts/fleet/audit.js`	the file/dir exists on disk
CLI flag	`--from-issues`	the literal appears in a shipped command/script source
Surface count	“18 universal commands”	the live gen-reference inventory count

Bare words (cap, todo, cdprov) are not checked — in prose they are indistinguishable from domain nouns, and repo aliases like cdprov resolve through shell config outside the repo. In-repo CLI entrypoints are still covered, via their scripts/… / bin/… paths.

  STALE  docs/explanation/workday-scenarios.md:84  /nonexistent-command   (no such command or skill)

FAIL — 1 stale reference(s) in docs/explanation/workday-scenarios.md.

How to fix a failure:

• Renamed/removed surface — update the reference in the page to the surface’s current name (or drop it).
• Deliberate forward-reference — when the page intentionally names a surface a sibling requirement will add (e.g. /update-journey, built by #846), declare it in an HTML comment in the page so the allowance is explicit and visible:

  <!-- drift-allow: /update-journey -->

  Remove the allowance once the surface ships.

Run it directly with make check-scenarios-drift (it is part of make check).

---

Repairing drift (prune & repair-fleet)

check-fleet reports drift; these repair it (#303).

cdprov --prune — per-project

Removes dangling symlinks under a project’s .claude/ that point into BoB (*/.claude/*, $BOB_SOURCE, $BOB_HOME). It is dry-run by default; pass --yes (or --apply) to delete. Each candidate is classified:

• removed-tooling — target under this machine’s BoB home/source but gone (e.g. the retired PM-file / sprint / requirements / risk items).
• cross-machine — an absolute target under a different home’s .claude (e.g. a repo provisioned on another box: /home/farm/.claude/...).
• other — any other dangling BoB symlink.

Safety guarantees (covered by make test-prune): prune never removes a resolving symlink, a real (non-symlink) file, or a foreign symlink (one whose target is not a BoB path). It is idempotent.

cdprov --prune            # dry run — preview what would be removed, by class
cdprov --prune --yes      # apply

make repair-fleet — every project

Walks projects.json and, per live project, runs cdprov --prune then (on apply) cdprov --refresh to re-link currently-declared items. Dry-run unless ARGS=--yes, and best-effort — a failure in one project warns and the run continues.

make repair-fleet            # dry run across the whole fleet (prune preview)
make repair-fleet ARGS=--yes # apply: prune + refresh every registered project
make check-fleet             # confirm remaining drift afterwards

The prune→refresh order matters: prune clears stale/cross-machine links first, then refresh recreates the items the manifest still declares.

---

JSON Schemas

Located in schemas/. Each defines the exact structure a config file must follow.

manifest.schema.json

For provisions/*.json files:

{
  "_meta": {
    "project": "string (required)",
    "path": "string | null",
    "stack": ["string"]
  },
  "skills": ["string (unique)"],
  "commands": ["string (unique)"],
  "agents": ["string (unique)"]
}

projects.schema.json

For projects.json:

{
  "_meta": {
    "description": "string (required)",
    "updated": "YYYY-MM-DD"
  },
  "projects": [{
    "name": "string (required)",
    "path": "string (required)",
    "group": "string (required)",
    "active": "boolean (required)"
  }]
}

settings.schema.json

For settings.json:

• permissions.allow / permissions.deny — string arrays
• hooks — keyed by event name (SessionStart, PreToolUse, PostToolUse, Stop, Notification, SubagentStop)
• Each hook entry has type: "command", command: string, optional timeout: integer
• statusLine — type + command
• enabledPlugins — map of string: boolean

---

File Layout

~/.claude/
├── Makefile                          # Entry point — run `make help`
├── schemas/
│   ├── manifest.schema.json          # Provisions manifest schema
│   ├── projects.schema.json          # Project registry schema
│   └── settings.schema.json          # Settings schema
├── scripts/checks/
│   ├── check-deps.sh                 # Dependency verification
│   ├── check-hooks.sh                # Hook script verification
│   ├── check-manifest-meta.sh        # Manifest identity (_meta.project / path, #1869)
│   ├── check-provisions.sh           # Manifest-vs-reality verification
│   ├── check-schemas.js              # JSON Schema validation runner
│   └── schema-validator.js           # Shared validator module (zero deps)
├── tests/
│   ├── test-checks.sh                # Integration tests (13 tests)
│   └── test-schemas.js               # Schema validator unit tests (34 tests)
└── .github/workflows/
    └── ci.yml                        # GitHub Actions — runs on push/PR

---

CI

.github/workflows/ci.yml runs on pushes and PRs to master:

1. Checkout repo
2. Setup Node.js 22 (the repo is ESM-first and relies on Node’s ES-module auto-detection)
3. Verify the runner toolchain (jq) and install a pinned PyYAML for the versioning tests
4. make check-deps check-schemas check-hooks check-intake (skip symlinks/provisions — no projects on a CI runner)
5. make test
6. make docs-check

CI skips check-symlinks and check-provisions because they validate local provisioning state — the cdi-created symlinks in a working copy — which doesn’t exist in a bare CI checkout (.claude/ symlinks are gitignored). Since #294 these checks default to the current repo (so make check in any working copy is no longer failed by drift in other registered projects); the cross-project sweep moved behind --all / make check-fleet. They remain local/dev gates by design.

Two lanes: source and docs (#1693)

CI is not the only gate, and it deliberately does not run on every push. The repo has two workflow lanes, split by path:

Lane	Workflow	Triggers on	Runs
Source	ci.yml	everything except docs/**, runbooks/**, README.md, CLAUDE.md	the checks, the full test suite, docs-check
Docs	docs-deploy.yml	exactly those doc paths (plus the docs-site tooling)	site build, flat-link verify, docs-check, deploy

A docs-only change therefore runs the docs lane alone rather than dragging a 240-target suite behind it, and make docs-check is what makes that safe: the doc naming, filename-convention, and anchor gates are prerequisites of that target, not separate CI steps, so they fire from either lane. Both lanes also carry a per-ref concurrency group with cancel-in-progress, so a burst of pushes bills one run rather than one per commit.

Two consequences worth knowing:

• Keep the two path lists in step. A path that is in ci.yml’s paths-ignore but not in docs-deploy.yml’s trigger paths is gated by nothing. make test-ci-lane-split asserts the containment and fails if it ever drifts.
• A few test targets read the real docs tree, and those do not run on a docs-only push — they run on the next push that touches source. No release can be cut in between, because auto-release.yml is itself filtered to the source surface. The site-breaking risks (build failure, orphan, drift) are all covered by the docs lane, which does run.

The release bot’s own commit is excluded from both lanes: its subject carries [skip ci] (see Versioning), since a version bump plus a regenerated changelog gives either lane nothing to decide.

---

Common Workflows

After cloning Bob on a new machine

cd ~/projects/bigbrain    # BOB_SOURCE
make check-deps           # verify tools are installed
make check-schemas        # verify configs are valid
make check-host           # this machine's BOB_HOME matches source (or: bob doctor)

After changing settings.json or a hook

make check-hooks check-schemas

After changing a provision manifest

make check-schemas check-provisions

After adding a new project

# Add to projects.json, then:
make check-schemas      # validates projects.json structure
make check-symlinks     # checks new project's symlinks
make check-provisions   # checks new project matches manifest

Before pushing changes to Bob

make ci     # runs everything

Investigating a project that feels broken

make check-symlinks     # find dangling symlinks
make check-provisions   # find manifest mismatches
cdprov --refresh /path/to/project   # fix it

---

Rendering the scenarios page (two-render split, #844)

The narrative workday scenarios page carries an introspection layer — the “friction margin” that is honest about where BoB’s seams are clumsy. That honesty makes a poor outsider-facing brochure, so the friction content is wrapped in HTML-comment markers and stripped or kept at render time, from a single source (no forked copies to keep in sync):

<!-- friction:TYPE -->   ...introspection-only content...   <!-- /friction -->

TYPE is one of pain-point / automation-opportunity / candidate-feature / section. Produce either render with scripts/render-scenarios.js:

# Outsider-facing: friction blocks removed, clean narrative.
node scripts/render-scenarios.js --mode showcase       [--out showcase.md]

# Maintainer-facing: friction content kept (markers stripped).
node scripts/render-scenarios.js --mode introspection  [--out introspection.md]

The transform is pure and deterministic — same input always yields the same output, and re-running it is idempotent. HTML comments (the friction fences, the authoring note, the drift-allow marker) are internal metadata and are stripped from both renders; only the friction content differs between them. The test-render-scenarios target proves showcase-strips / introspection-keeps / idempotent.

Design Principles

• Zero dependencies — only node, jq, git, make (all pre-installed or one brew install)
• Every check exits 0 (pass) or 1 (fail) — no ambiguity
• Actionable output — every failure includes a fix instruction
• Idempotent — safe to run any target as many times as you want
• Fast — full make check completes in under 5 seconds
• Composable — run individual targets or combine them: make check-deps check-hooks