Docs-Site Lifecycle

Docs-Site Lifecycle (#153, #689)

Provision and operate the engine-pluggable docs site. Extracted from the dev lifecycle reference.

Docs Site Lifecycle

Every BoB-provisioned project can host a docs site that reads its own docs/, README.md, and CLAUDE.md. The lifecycle is orthogonal to the dev environment lifecycle but follows the same declarative pattern.

The tooling is engine-pluggable (#689). The engine field in docs-site.json selects the static-site generator:

Engine	Value	Notes
VitePress (default)	`vitepress`	Flat `.html` with native `.md` link rewriting — correct under flat Cloudflare-Pages serving with no post-build step. Folder-autogenerated Diátaxis sidebar (`vitepress-sidebar`), built-in local (MiniSearch) search.
Astro Starlight (alternative)	`starlight`	`build.format: 'file'` + `trailingSlash: 'never'` for the same flat serving. Folder-autogenerated nav, Pagefind search, richest theming.

Both render the same docs/ taxonomy and resolve cross-directory links correctly under flat serving with no post-build link rewriting. Quartz — and its fix-links.js hack — was retired in #689 (building on the #487 SSG research).

Accepted regression: Quartz’s graph view and automatic backlinks have no equivalent in either engine; [[wikilinks]] need a plugin or a one-time conversion.

Node requirement: the Starlight engine (Astro 7) needs Node ≥ 22.12; VitePress is fine on Node ≥ 18. CI pins Node 22 in docs-deploy.yml for this reason.

Every sub-command runs two ways — as a slash command in a Claude Code session, or from a terminal by invoking its script directly. The table pairs them; the examples further down show the slash form, but each has the terminal twin listed here.

Slash command	Terminal	Purpose
`/docs-site init`	`~/.claude/scripts/docs-site/init.sh`	Scaffold `.docs-site/<engine>/` for the resolved engine + a default `docs-site.json` (if missing). Idempotent on re-run. `--engine` forces an engine.
`/docs-site build`	`~/.claude/scripts/docs-site/build.sh`	Generate reference pages from command/skill/agent frontmatter (see below), then sync the `docs/` taxonomy + run the engine build. Flat output at `.docs-site/public/`. `--engine` overrides.
`/docs-site dev`	`~/.claude/scripts/docs-site/dev.sh`	Sync content + engine dev server with hot reload. Binds the project’s docs port from the port ledger (`base+3`), so it never collides with the app or another project. `--port` overrides; `--engine` overrides the engine; `--print-port` resolves and prints the port without starting a server.
`/docs-site deploy`	`~/.claude/scripts/docs-site/deploy.sh`	Build, then publish to the configured target (`cloudflare-pages`, `static`, `custom`). The primary engine deploys to `deploy.project`; a comparison engine deploys to `<deploy.project>-<engine>`.
`/docs-site draft`	`~/.claude/scripts/docs-site/draft.sh`	Build + publish a PR-independent preview from any branch into an isolated URL namespace (needs `draft.enabled` — see the draft/preview section below).
`/docs-site clean`	`~/.claude/scripts/docs-site/clean.sh`	Drop build artifacts (preserves installed deps). `--hard` removes the whole `.docs-site/` workspace.

Per-project config (docs-site.json at project root):

{
  "engine": "vitepress",
  "title": "Bodmail Docs",
  "description": "Documentation for the Bodmail email platform",
  "base_url": "https://docs.bodmail.app",
  "sources": ["docs/", "README.md", "CLAUDE.md"],
  "exclude": ["docs/archive/**", "docs/audits/**"],
  "deploy": { "target": "cloudflare-pages", "project": "bodmail-docs" }
}

base_url drives the serve path. The engine base is the path component of base_url, normalised to /…/ — / for a root-served Pages project (the BoB default, empty/host-only base_url), or e.g. /docs/ only if the site is genuinely served under that subpath. A base that doesn’t match where the site is actually served makes every asset 404 (an unstyled page). The flat-link gate (make docs-verify) runs at this same derived base.

How each declared source lands (#1261). docs/ is special-cased: its contents sync to the content root, so the Diátaxis taxonomy sits at the top (docs/how-to/x.md → /how-to/x). Every other declared directory publishes as a named section keeping its own directory (runbooks/x.md → /runbooks/x), and a declared file publishes at the root under its basename (CLAUDE.md → /CLAUDE). README.md is the home page when the content provides no index.md of its own.

Cross-root relative links are fixed at sync time (#1468). Because docs/ loses a level and a named section does not, a relative link between the two resolves on GitHub but would land at the wrong site path. The sync’s rewrite pass (rewrite-escapes.js) detects in-scope links whose synced location shifted and rewrites them on the synced copy to the site-correct relative path — so author every link for the repo (docs/index.md → ../runbooks/index.md, docs/** → ../../CLAUDE.md, a runbook → ../docs/how-to/x.md) and the site gets the corrected form. As a backstop, verify-links.js --flattened docs fails any surviving docs/-first-segment link as an off-by-one instead of skipping it as external.

runbooks/ is published (#1447). BoB declares runbooks/ as a source, so the universal runbooks are a first-class site section rather than repo-only files. The reasoning, since it is the kind of thing that gets re-litigated: a runbook is a reader-facing procedure (Purpose / When to use / Prerequisites / Steps), /runbook-gen exists to produce more of them, and a generated surface nobody can read is a coverage gap. Three consequences worth knowing:

Promotion is publication. runbooks/.drafts/ is git-ignored and excluded from the site; promote.js moving a draft into runbooks/ is what puts it on the web. See Generate a Runbook.
Membership is derived. runbooks/index.md is a managed section index (scripts/docs-site/lib/section-index.js), so a promoted runbook joins the index — and stops being an orphan — by existing. runbooks/**/*.md is in the doc-keeper scan_paths, so runbooks are governed like any other published doc.
A runbook edit republishes. runbooks/** is a path trigger on .github/workflows/docs-deploy.yml.

A project that does not want its runbooks published simply omits runbooks/ from sources — the directory still works as agent-loadable scoped context.

Docs-currency policy (publish_on, draft) — #759. Two optional keys declare when docs publish:

publish_on (merge | production-deploy, default merge). merge publishes on push to the main branch — the existing behavior, so omitting the key is a no-op. production-deploy holds the production publish until the app’s production deploy has succeeded, for projects where merge ≠ production: deploy.sh gates the publish on a prod-deploy-ok signal (--prod-deploy-ok or DOCS_SITE_PROD_DEPLOY_OK=true) and otherwise skips the publish (a hold, not a failure).
draft ({ enabled, path_prefix }, default { enabled: false, path_prefix: "drafts" }). Declares the draft/preview mechanism the PR-independent preview workflow consumes — when enabled, a build from any branch can publish a preview into an isolated URL namespace (/<path_prefix>/<branch>/) without touching the live site.

Both are validated by schemas/docs-site.schema.json, and this repo’s docs-site.json is checked against it in make check (check-schemas).

Sidebar order and search scope (nav) — #2607. By default the VitePress sidebar is built from the folder tree: sections in alphabetical order, every section expanded. An optional nav block sets the order instead:

"nav": {
  "pinned_label": "Start here",
  "pinned": [
    { "text": "Docs home", "link": "/" },
    { "text": "bob CLI reference", "link": "/reference/cli" }
  ],
  "order": ["tutorials", "how-to", "reference", "explanation", "runbooks", "research", "audits", "archive"],
  "collapsed": ["research", "audits", "archive"],
  "search_exclude": ["archive/", "audits/", "research/"]
}

order lists the top-level entries in the order they appear. Each key is a section folder (how-to, runbooks) or a page at the site root without its extension (vision). Entries you don’t list follow the listed ones, in the default order.
collapsed lists the sections that start collapsed. They stay in the sidebar and can be expanded.
pinned adds a group above every section, titled by pinned_label (default Start here), for the pages you open most. Each link is a site path starting with /.
Pages within a section follow the order of the links in that section’s index.md, so to move a page up, link it earlier on the index page. To set the order explicitly instead, list the pages under sections, for example "sections": { "how-to": ["warp-drive-cli", "troubleshooting"] }. Pages you don’t list follow the listed ones.
search_exclude lists path prefixes to leave out of local search. The pages are still built and still reachable by link; they just don’t come up as search results. To exclude a single page, add search: false to its frontmatter.

Without a nav block the sidebar is built as before. The block applies to VitePress only; the Starlight engine ignores it. The code is templates/docs-site/vitepress/nav.mjs, which /docs-site init and /docs-site build copy into the workspace alongside config.mjs. The tests are in tests/test-docs-site-nav.js.

Adopting in a project:

cd /path/to/project && /docs-site init — scaffolds .docs-site/<engine>/ and a default docs-site.json.
Edit docs-site.json — set the engine, title, base URL, and deploy target.
/docs-site dev — preview at the printed URL (the project’s ledger docs port; run /docs-site dev --print-port to see it).
/docs-site deploy when ready.

Trying the other engine, ad hoc (no config change):

Slash command:

/docs-site build --engine starlight
/docs-site dev   --engine starlight

Terminal:

~/.claude/scripts/docs-site/build.sh --engine starlight
~/.claude/scripts/docs-site/dev.sh   --engine starlight

Flat-serving link gate (make docs-verify): builds both engines against the real docs/ and asserts every in-docs cross-directory link resolves under flat Cloudflare serving with no rewriting (#689 AC-04; scripts/docs-site/verify-engines.sh + verify-links.js). Links pointing outside the published docs (repo source) are skipped, not failed.

Out-of-scope link rewrite (#1393): a relative link that resolves on disk but escapes the declared content roots (registry/…, apps/…, skills/…) is valid on GitHub yet 404s on the published site. Every sync runs scripts/docs-site/lib/rewrite-escapes.js over the synced copy — such links become <repo>/blob|tree/<branch>/<path> GitHub URLs (remote/branch from origin; anchors preserved; fenced and inline code untouched), so sources stay GitHub-idiomatic while the published site never links into content it doesn’t carry. The companion guard (rewrite-escapes.js check, wired into make docs-check) fails only on an escaping link whose target doesn’t exist on disk — that one 404s everywhere and needs a hand-fix.

Cross-root link rewrite (#1468): the same pass also fixes in-scope links whose relative depth changes when the sources sync (docs/ flattens to the root; named sections and root files do not): the link is rewritten on the synced copy to the site-correct relative path, using the inverse of the same source maps. This runs even with no GitHub remote — it is a relative-path fix, not a GitHub URL rewrite. The flat-serving gate passes --flattened docs to verify-links.js, so a leaked repo-rooted path (docs/… first segment) fails as an off-by-one rather than being skipped as external.

Universal portability: the scripts read --root, $PROJECT_ROOT, or cwd — they never hardcode BOB_SOURCE. The .docs-site/<engine>/ workspace is entirely generated from templates/docs-site/<engine>/ + the synced docs/, so the whole directory is gitignored.

Auto-deploy on doc changes (#155). A GitHub Actions workflow at .github/workflows/docs-deploy.yml rebuilds, verifies, and publishes the site whenever a published doc source changes (docs/**, README.md, CLAUDE.md, docs-site.json, the docs-site scripts, or the engine templates). On master push it deploys both engines for a live A/B; on PRs it deploys previews and posts the URLs back as a sticky comment.

The pipeline is:

checkout → cache .docs-site/{vitepress,starlight}/node_modules
        → make docs-check                 (doc-governance job — reports, does NOT block deploy)
        → verify-engines.sh               (build + flat-link-verify BOTH engines)
        → build vitepress → wrangler pages deploy → bigbrain-docs            (primary)
        → build starlight → wrangler pages deploy → bigbrain-docs-starlight  (comparison, best-effort)
        → verify-deploy.sh                (fetch the published root + changed pages)
        → comment preview URLs on the PR

Post-deploy verification (#1446). A green wrangler pages deploy says the upload succeeded, not that the site publishes — the build can emit nothing, the output directory can be wrong, a routing change can 404 every page, and none of that turns the run red on its own. So after a production deploy the workflow fetches the site back:

~/.claude/scripts/docs-site/verify-deploy.sh --project bigbrain-docs \
  --doc docs/how-to/foo.md          # routes are derived from changed doc sources

Checks the site root plus the pages the pushed docs publish to (docs/how-to/foo.md → /how-to/foo, index.md → its directory, README.md → the home page). A source that publishes no page is skipped, not failed.
Follows redirects — Pages legitimately 308s clean URLs, so only the final status is the verdict.
Retries with backoff to absorb propagation delay, bounded by --attempts (6) and --max-wait (180s), so a dead deploy fails in bounded time.
A failure fails the run, which is what makes notify-failure fire.
Skipped when DOCS_SITE_DEPLOY_ENABLED != 'true' — there is nothing published to verify, and the whole deploy job is gated on that flag.
Works unchanged for a comparison engine: it verifies whatever project name that matrix leg resolved (<project>-<engine>). Only the primary engine is verified at a configured base_url.

deploy.sh runs the same check after a manual publish; pass --skip-verify to publish without it.

Blast radius: the two gates are not equal (#1444). A doc that nothing links to and a site that will not build are different failures, and only one of them should stop publishing:

Job	Catches	Deploy needs it?	A red run on master means
`gate`	Build failure, flat-serving links that don’t resolve	Yes	Nothing publishes. The last-known-good deployment stays served and goes stale until fixed.
`doc-governance`	`make docs-check` — orphan / root-placement / freshness	No	The site still republished. The offending page is live but hard to find; the failure is scoped to it rather than freezing every page.

Both turn the workflow red, both file (or bump) the docs-deploy-failure issue, and on a PR both block the merge via branch protection.

The alert clears itself (#1445). A green master run closes every open docs-deploy-failure issue, commenting the run URL and commit SHA. Green means the gate passed, doc governance passed, and deploy either published or was legitimately skipped (publishing disabled — the pipeline is healthy, it just isn’t publishing). A green run with no open issue is a silent no-op. So an open docs-deploy-failure issue means the pipeline is red right now — before this, one stayed lit for twelve days of successful runs, which is exactly how an alert gets ignored. What changed is that a findability finding no longer holds every other page hostage — a single orphan used to freeze the entire site’s republishing until a human noticed. The primary defence against that class is now local and earlier anyway: hooks/doc-orphan-gate.sh refuses the commit that would land the orphan, naming the index to edit.

Required repo configuration:

Secret CLOUDFLARE_API_TOKEN — Cloudflare API token with Pages:Edit scope.
Secret CLOUDFLARE_ACCOUNT_ID — the Cloudflare account UUID.
Variable DOCS_SITE_DEPLOY_ENABLED=true — feature flag that gates the wrangler steps. Until set, the workflow still builds, verifies, and runs the drift gate (catching breakage) but skips the deploy.
A one-time bigbrain-docs-starlight Cloudflare Pages project for the comparison build — the Starlight deploy is best-effort, so a missing project never blocks the primary production deploy. See #187 for the Cloudflare connection step.
Secret BOB_CHECKOUT_TOKEN — on non-BoB projects only: a fine-grained PAT (Contents: Read-only on the BoB repo) so CI can check out the private BoB tooling. Without it the gate degrades gracefully — the BoB checkout is non-fatal and engine verification skips with a loud ::warning naming the secret; the deploy job stays strict (#1265). docs-site init provisions the secret (or files a todo), and ~/.claude/scripts/fleet/distribute-secret.sh status|distribute|reconcile handles the whole fleet in one pass — see the docs-site skill for details.
Variable BOB_REF — on non-BoB projects only: the ref of the BoB repo the docs-site tooling is read from (default master). It should be a tag or a commit, so the job that holds the deploy credentials runs a version someone chose rather than whatever the branch holds that day. The BoB checkout does not keep BOB_CHECKOUT_TOKEN on disk (persist-credentials: false, #3648).

NPM script aliases live in the root package.json so users can run npm run docs:build / docs:dev / docs:deploy from anywhere in the repo.

Draft / preview, PR-independent (#760)

/docs-site draft builds and publishes a shareable preview from any branch without opening a PR — the counterpart to /docs-site dev (local hot-reload). It is keyed to the current git branch (--branch overrides), and the mechanism is read from the draft config block (#759), so it must be enabled per project:

Slash command:

/docs-site draft

Terminal:

~/.claude/scripts/docs-site/draft.sh

{ "draft": { "enabled": true, "path_prefix": "drafts" } }

Cloudflare Pages → a per-branch preview alias (wrangler pages deploy --branch <slug>), reachable independent of any PR.
Static target → the build is staged at .docs-site/<path_prefix>/<branch>/, leaving the production output (.docs-site/public) untouched.

docs-site dev (local hot-reload) is unchanged and remains the inner-loop tool; draft is the shareable preview. PR-preview deploys from docs-deploy.yml stay a best-effort bonus when a PR happens to exist — never the mechanism, so previewing WIP docs never requires a PR.

Build provenance on every page (#2620)

The site deploys from the default branch on every push, so it usually runs ahead of the newest release, and a machine on the stable channel may not have a command a page describes. Every page therefore says which build it describes and how current it is, in one line above the content:

Build master @ 2d60e31 · unreleased — 37 commits past v1.274.1 · Last reviewed Sep 3, 2026 · Last changed Sep 18, 2026 · Grade aging (aging >60d, stale >120d) · Source / History

Build: the branch and short SHA the site was built from. The hover title shows the full git describe version and points to bob doctor, whose version line gives the version your machine runs. The browser cannot read that version itself.
Release state: matches v1.274.1 when the build is a release tag, a highlighted unreleased — N commits past v1.274.1 when it is ahead of one, or version unknown with the reason in its title.
Last reviewed / Last changed: the authored last_reviewed (or generated_at on a generated page) and the source file’s last commit. Last changed is shown only here; the VitePress doc footer no longer shows it.
Grade: fresh / aging / stale / frozen, from the staleness policy make docs-check uses (documentation standard §3.4), with the page type’s thresholds. It is graded at build time.
Source / History: the page’s source file, and its commit history, at the built commit. The repo comes from the declared upstream (scripts/bob/upstream.json, #2441) when the project declares one, else the origin remote. Links use the forge’s own path shape (GitHub, GitLab, Gitea/Forgejo/Codeberg).

A preview build gets a banner above the strip, Preview of <branch> — not a published build. That covers /docs-site draft, and a PR build in docs-deploy.yml: both set DOCS_PREVIEW_BRANCH. Every page, including the home page and pages with no frontmatter, also ends with a footer line: Built from v1.274.1-37-g2d60e31 · 2d60e31.

The build stamp. The same facts are served at /version.json as docs-build/1 (schemas/docs-build.schema.json):

{ "schema": "docs-build/1", "version": "v1.274.1-37-g2d60e31", "version_reason": null,
  "sha": "2d60e31", "branch": "master", "built_at": "2026-09-18T14:02:11Z",
  "release_tag": "v1.274.1", "commits_since_release": 37, "preview": null,
  "source": "git-describe", "shallow": false, "forge": { "web": "…", "blob": "…/blob/{sha}/{path}", "history": "…" } }

scripts/docs-site/lib/build-stamp.js resolve --root . prints it without building. The stamp’s version comes from an optional version block in docs-site.json:

{ "version": { "source": "git-describe", "tag_pattern": "v[0-9]*" } }

source is git-describe (the default) or none. tag_pattern names release tags, so nightly-* and archive/* tags never count as a release.

Honest degradation. In a shallow clone, or a repo with no tag that matches, the stamp keeps the SHA and build time, sets version to unknown, and says why in version_reason. The strip and footer show that reason. A shallow clone is never trusted for a version, because a newer tag that was not fetched would make the answer a guess. docs-deploy.yml checks out with fetch-depth: 0 so production builds resolve.

Engines. Both engines serve /version.json, and make docs-verify checks it. The strip and footer are VitePress only (the primary engine), and make docs-verify checks that they render. Known gap: Starlight shows its own last-updated and review dates but has no provenance strip or build footer yet.

Generated reference (#758)

build.sh runs scripts/docs-site/lib/gen-reference.js before the engine sync, harvesting the frontmatter of every command, skill, and agent surface (universal and provisioned registry/ equivalents) into per-category reference pages under docs/reference/generated/ — commands.md, skills.md, agents.md, and an index.md. For a tooling project these commands are the API surface, so no separate API harvest is needed.

It also emits cli.md, the shell CLI surface (#2061). That one is harvested differently by necessity: shell commands are declared in scripts/bob/commands.json, not in frontmatter, so it is read through the help engine that renders bob help --json (bob-help/1). The same renderer produces the  block body that scripts/doc-keeper/sync.js splices into CLAUDE.md, the README and the glossary, and that make check-cli-table gates — one renderer, several destinations, so the docs-site page and the committed tables cannot disagree. A project with no bob routing table simply gets no CLI page; generation degrades rather than failing.

Since (#2622). Both CLI pages show the release each command first shipped in: a Since column in the generated cli.md, and a Since: value per command in the one-page docs/reference/cli.md. scripts/bob/since.js derives the values from git. It finds the first commit that declared each command in scripts/bob/commands.json, then the lowest release tag that contains that commit. A command no release includes shows unreleased. A shallow clone or a repository with no release tag shows unknown, and the page names the reason. These are the same rules the build stamp above follows. build.sh renders docs/reference/cli.md again on every build, so the published values are always current. The committed copy is checked with the values masked out, so a new release never makes it fail the drift check.

The generated pages are a build artifact, not hand-authored content:

Reproducible / idempotent — same sources produce byte-identical output; the output dir is wiped each run, so a removed or renamed surface drops out with no orphaned page left behind.
Machine-generated — each page carries generated: true in its frontmatter and a MACHINE-GENERATED banner; the path is git-ignored and excluded from the manual-edit doc-drift checks (scripts/doc-keeper/rules.json), so it never trips drift gates.
100% coverage — every command/skill/agent surface appears on a generated page, and every bin/ entrypoint is a row in the bob routing table (an unmapped one is a coverage gap: the generated CLI table would omit a command the user can actually run). gen-reference.js --check-coverage prints the coverage JSON and exits non-zero if any surface is unlisted; make docs-reference-check runs it as a gate.

Projects with no command/skill/agent directories skip generation entirely.

Drift report on the site (#156)

Every /doc-audit run writes two artifacts to docs/audits/:

doc-audit-YYYY-MM-DD.{md,json} — date-stamped historical record
latest.{md,json} — canonical aliases the docs site reads

The site (when built via /docs-site build) consumes docs/audits/latest.md for a “Docs last verified” badge on the landing page and an “Audits” entry in the nav. latest.json is exposed as a downstream API endpoint at /audits/latest.json. See docs/audits/ for the canonical-path contract and history convention.

The nav + landing-badge component wiring lands as a follow-up — the data plumbing (latest aliases + history) is in place.