Agency Agents Evaluation

Spike: Agency Agents evaluation

Eval spike for #1234, feeding the “AI agency” capability (#1030) and the multi-harness cluster (#498). Exploratory notes, not normative.

Subject: msitarzewski/agency-agents (~145K stars at review time; the issue cited it as “msitarzewski/agency” at 131K — the repo has since grown and the slug is agency-agents). A collection of 250+ agent persona files organized into ~20 “divisions” (engineering, design, testing, product, project-management, marketing, sales, security, specialized, …), plus shell tooling (convert.sh / install.sh) that transforms and installs them into 14+ harnesses (Claude Code, Cursor, Codex, Gemini CLI, OpenCode, Windsurf, Aider, …) and a native desktop app (agencyagents.app) that does the same with one click.

What a persona file looks like

Each agent is one Markdown file with light frontmatter (name, description, color, emoji, vibe) and a long system-prompt body with a consistent internal skeleton — the “voice + workflow + deliverables” structure:

Section	Content	Example (Reality Checker)
Identity & memory	Role, personality traits, remembered failure patterns	“Skeptical, thorough, evidence-obsessed, fantasy-immune”
Core mission	2–4 named objectives with bullets	“Stop fantasy approvals”, “Require overwhelming evidence”
Critical rules	Non-negotiable constraints, phrased as behavior	“Default status is NEEDS WORK until overwhelming proof says otherwise”
Mandatory process	Numbered steps, often with literal shell commands	“STEP 1: Reality Check Commands (NEVER SKIP)” + a `bash` block
Deliverables	Named outputs and evidence formats	screenshots dir, `test-results.json`, a certification verdict
Success metrics / communication style	How it reports, what counts as done	honest C+/B- grades are normal; first pass is auto-incomplete

The differentiator over “you are a helpful assistant” templates is that the persona carries a concrete process with evidence obligations, and the personality exists to enforce the process (the Reality Checker’s skepticism is a quality gate, not flavor text).

Personas that map to BoB workflows

Filtering ~250 agents down to the ones aligned with what BoB actually orchestrates (dev tooling, docs, QA, PM — not marketing/sales/China-platform divisions, which are out of scope for BoB entirely):

Agency persona	Maps to BoB surface	Overlap
Reality Checker (testing)	warp-drive `qa` phase; swarm result gate (#579) / grader (#580)	High — same “evidence before certification” job; BoB’s is mechanical, theirs is persona-enforced
Evidence Collector (testing)	`webapp-testing` skill (Playwright screenshots)	High — BoB has the tooling, not the persona discipline
Code Reviewer (engineering)	`registry/agents/code-reviewer.md`, `/code-review`	High — already covered
Test Automation Engineer (testing)	`unit-test-generator`, `qa-strategist` agents	Medium
Codebase Onboarding Engineer (engineering)	`/where` skill, `Explore` agent	Medium — read-only orientation job
Technical Writer (engineering)	`doc-keeper` agent + doc-audit/doc-sync skills	Medium — BoB’s is drift-focused, theirs authors narrative (a BoB non-goal: narrative stays human-authored)
Minimal Change Engineer (engineering)	warp-drive chunk discipline; `/simplify`	Medium — “minimum-viable diff, no scope creep” as a persona
Senior Project Manager (project-management)	`software-project-manager` skill, `project-manager` agent	Medium — “realistic scoping, spec→task conversion” echoes `/flightplan` + chunking
Multi-Agent Systems Architect (engineering)	the #1030 orchestration capability itself	Low-direct, high-conceptual — a persona about designing agent systems
Tool Evaluator (testing)	this very spike’s job; `/research` skill	Low — process reference
Whimsy Injector (design)	nothing in BoB	None — cited for contrast: persona-first, process-light

Structural comparison with BoB’s agent framework

BoB defines agents (3 universal in agents/, 22 specialists in registry/agents/) per the agent-builder conventions, enforced by make check (scripts/checks/check-agents.js, #330):

Dimension	Agency Agents	BoB
Selection contract	`description` is a one-liner tagline; humans browse a README roster and install what they like	`description` is the machine trigger: ≥15 words of trigger phrasing + 2–4 `<example>` blocks, because auto-delegation matches on it — enforced by CI
Tool scoping	None — personas assume whatever the harness allows	Mandatory least-privilege `tools:` list; `Agent` tool banned in subagents (fan-out guard) — enforced by CI
Model routing	None	Mandatory explicit `model:` per agent for cost predictability — enforced by CI
Voice / personality	Rich, deliberate, load-bearing (skepticism-as-quality-gate)	Mostly flat “You are a [role] with expertise in…”
Process	“Mandatory process” with numbered steps and literal commands inside the persona	Process lives mostly outside the agent — in skills, hooks, and the warp-drive state machine; agent bodies describe expertise more than procedure
Deliverables	Named per persona (evidence files, verdict formats)	Named per workflow (chunk reports, session summaries, typed check results) rather than per agent
Distribution	`convert.sh` → per-harness formats; `install.sh` / native app installs into 14+ tools	`cdprov` + provisioning manifests, symlinks from a deployed registry — Claude-Code-only today (#498 targets Codex/Gemini/Cursor)
Governance	Lint scripts (`lint-agents.sh`, originality/division checks)	Schema validation + `make check` least-privilege hygiene + provisioning manifests as declared state

The structural insight: the two collections optimize opposite halves of the same file. Agency invests in the body (voice, mandatory process, evidence obligations) and ignores the plumbing (no tool scoping, no model routing, no machine-readable triggers). BoB invests in the frontmatter and governance (triggers, least-privilege, cost, declared provisioning) while its bodies are comparatively procedural-thin — because BoB deliberately keeps process in version-controlled skills/hooks/state machines rather than inside a persona’s prose. Adopting Agency’s body discipline does not require importing its plumbing gaps.

Candidates to adapt (not adopt verbatim)

Three personas earn a prototype, in priority order. Each would be rewritten to BoB conventions (trigger-phrased description with examples, least-privilege tools:, explicit model:) while keeping Agency’s body discipline — the named mission, the non-negotiable rules, the mandatory evidence steps.

Reality Checker → a qa-verifier agent. The strongest fit. Warp-drive’s qa phase and the swarm gate/grade loop already demand evidence before promotion, but the judgment half (“is this evidence actually convincing?”) is unpersonified. A skeptic whose default verdict is NEEDS WORK, who treats perfect scores as red flags and cross-checks claims against artifacts, is a direct upgrade to the existing qa-strategist — and its “first pass is auto-incomplete” stance matches the grader’s revise loop.
Minimal Change Engineer → a scope-discipline overlay for warp-drive coding. “Fix only what’s asked, minimum-viable diff, no scope creep” is exactly warp-drive’s chunk contract (#992 scope, serial-only pinch points). As a persona block injected into the coding phase (or folded into team-implementer), it turns a state-machine rule into enforced behavior.
Codebase Onboarding Engineer → harden the Explore/where path. Read-only, trace-code-paths, state-facts-only. BoB has the tooling (/where, the Explore agent); the persona adds the discipline (never speculate, cite file:line for every claim) that makes orientation output trustworthy.

Worth stealing as a pattern rather than an agent: the “Critical Rules / Mandatory Process” body skeleton. The agent-builder skill’s anatomy (Your Expertise / Your Process / Output Format) could name evidence obligations the way Agency does — commands the agent must run before it may conclude. That is a one-line addition to the authoring conventions with outsized effect on agent reliability.

The installer approach — relevance to #498

Agency’s distribution pipeline is the part most relevant beyond agent content. Its convert.sh treats the persona .md as a canonical source format and mechanically derives per-harness artifacts (Claude Code agents/*.md, Cursor .mdc rules, Codex TOML, Gemini/OpenCode/Qwen variants, single-file CONVENTIONS.md for Aider…); install.sh then auto-detects installed tools and copies per-tool output into their config dirs, with division/agent filtering and dry-run. The native app wraps the same operation with auto-update.

That is structurally the same shape as the multi-harness cluster (#498, adapters #501/#503): one source of truth + per-harness adapters at the provisioning boundary. Findings worth feeding in:

Convert-then-install split. Generation (convert.sh → integrations/<tool>/) never touches user config; installation is a separate, targeted copy. That maps cleanly onto BoB’s deploy (source → BOB_HOME) vs provision (links into projects) split — the adapters belong at the cdprov link/copy step, not in the registry items themselves.
Auto-detection + filtering. --tool / --division / --agent / --dry-run and interactive detection of which harnesses exist are the right UX; cdprov manifests already express the filtering half declaratively.
Per-harness capability caps are real. The OpenCode ~119-agent runtime limit (installer warns when a selection exceeds it) is a concrete example of why #498’s adapters need per-harness constraints, not just format translation.
Not worth importing: the native app itself. BoB’s surface is cdprov and manifests; a GUI installer solves a discovery problem BoB doesn’t have.

Verdict: adapt (selectively) — not adopt, not pass

Not adopt: wholesale import fails BoB’s CI on every file (no tools:, no model:, tagline descriptions) and would bury 25 curated agents under 250 uncurated ones — the opposite of the #1132 minimal-core direction. License is MIT, so reuse is unencumbered; volume and convention mismatch are the blockers, not legality.
Not pass: the body discipline (personality-as-quality-gate, mandatory evidence steps, named deliverables) is a real gap in BoB’s comparatively procedural-thin agent bodies, and the convert/install pipeline is a working reference implementation of exactly what #498 plans to build.
Adapt: (a) prototype the three candidates above to BoB conventions; (b) extend agent-builder’s anatomy with a “critical rules + evidence obligations” section; (c) reference convert.sh/install.sh as prior art in the #498 adapter design. Each is small, independent, and reversible.

For #1030 (“AI agency” capability): this spike supports approving the capability, scoped to the adapt list — persona-enforced quality gates layered on BoB’s existing orchestration, not a 250-agent roster import.