Skill evals

A change to a skill is accepted only when it measurably helps. Each item that has a suite is run twice per case, once with the item loaded and once without it. The change passes only when the score with it is strictly higher than the best score recorded for that model, higher than the score without it, and its trigger rate does not drop.

BoB builds no eval harness of its own. It wraps the platform’s, claude plugin eval (Claude Code 2.1.272 or later; see the plugin evals docs). The suites live in BOB_SOURCE. Three things use them:

bob u exam uses them to examine an item before it is disposed (#2903).
The skill proposer (#2391) uses them to validate a proposed edit.
wiki/skill-impact.md keeps every outcome.

Layout

evals/
  <item>/                     one suite per skill, command or agent, named as the item
    baseline.json             the best score recorded per model (skill-eval-baseline/1)
    <case>/                   one directory per case
      prompt.md               the prompt; frontmatter sets max_turns, allowed_tools, tags …
      case.yaml               optional: the same keys in YAML (schema_version "1.1")
      graders/
        <grader>.md           one grader per file; frontmatter type: regex | tool_used | tool_order | file_exists | llm | baseline

The item is found under skills/<item>/, commands/<item>.md, agents/<item>.md, or their registry/ twins. A suite needs at least one case. Every case needs at least one grader.

Authoring a case

Make the case directory: evals/<item>/<case>/.
Write prompt.md. The frontmatter sets the run and the body is the prompt. An eval run cannot use gated tools such as Bash, Write or Edit unless the operator grants them. A prompt should therefore carry its own context (paste the diff, the file, the error) and ask for an answer the graders can read:
```
---
description: A commit request with the diff in hand.
tags: [trigger, smoke]
max_turns: 6
allowed_tools: [Skill, Read, Glob, Grep]
---

I fixed a crash … The staged diff is: … Commit it. You cannot run commands
here, so give me the exact commit message you would use.
```
Add graders under graders/, one file each. Score what the item teaches, which is the behaviour the no-item arm should get wrong:
```
---
type: regex
pattern: '^\W{0,3}(feat|fix|docs|refactor|perf|test|build|ci|chore|revert)($[a-z0-9._/-]+$)?!?: \S.{0,70}$'
flags: m
---
```
```
---
type: llm
---

The proposed commit message says WHY the change was made, not only what the diff does.
```
JavaScript regexes take no inline flags. (?i) and (?m) make the grader throw, so use flags: instead. An llm grader passes on 2 of 3 judge votes, and its judge is haiku unless the run passes --judge-model.
Add the trigger pair (see the next section).
Run the suite once on a cheap model before you record anything: bob eval run <item> --runs 1 --model haiku --max-cost-usd 0.2.

Trigger cases (every suite)

BoB’s weakness is triggering more than execution. A skill that would help often never fires (the session ledger’s no-trigger records, bob tour). So every suite carries a pair of trigger cases:

fires: a prompt where the item should fire, with a grader pinned to the item by input_match:
```
---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?commit"'
---
```
quiet: a prompt where it should not fire. min: 0 and max: 0 assert that it never fires, and arm: both scores the grader on both arms:
```
---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?commit"'
min: 0
max: 0
arm: both
---
```

Under the with/without ablation the platform treats a tool_used: Skill grader as a with-only indicator, which does not count toward the score. run.js reports these graders separately as trigger_rate: the share of all tool_used: Skill results on the with arm that passed. The share includes the quiet cases, so a skill that fires everywhere loses as much as one that never fires.

bob eval run (scripts/evals/run.js)

bob eval run commit --model claude-sonnet-5-5 --runs 3 --out /tmp/commit.json

run.js assembles a throwaway plugin in a temp directory. The plugin holds a .claude-plugin/plugin.json (a target without a manifest silently runs as a single arm), the item linked in at its place, and a copy of the suite (baseline.json and any old results/ are left out). It then runs:

claude plugin eval <plugin> --ablation with-without --json <file>
  --max-cost-usd <ceiling> --trust-plugin --no-publish [--model M] [--runs N]

It runs from the current directory: bob u exam runs it from the consuming project. It reduces the platform’s result to skill-eval/1 and validates the document against schemas/skill-eval.schema.json before printing it.

Field	Meaning
`with`, `without`	Mean case score with the item loaded and on the no-item arm (0–1)
`delta`	`with - without`
`verdict`	`improved` (delta > 0) · `no-signal` (0) · `regressed` (< 0)
`trigger_rate`	Share of `tool_used: Skill` results that passed on the with arm; `null` with no trigger grader
`cost_usd`, `max_cost_usd`	What the run cost, and the ceiling it was held to
`partial`	`true` when the ceiling stopped it before every case ran
`model`	`--model`, else the session default (`$ANTHROPIC_MODEL`, else BOB_HOME’s `settings.json` `model`), else `default`
`run`, `date`	`eval-<item>-<UTC stamp>` and the day the run started
`cases[]`	The same numbers per case

Cost ceilings are mandatory. Each case runs a full claude child on both arms, on the operator’s credential. The ceiling is --max-cost-usd, else _workflow.eval_max_cost_usd (default 2), and 0 or less is refused. The platform checks the ceiling before each run, so an overrun is bounded to the run in flight.

Exit codes: 0 the eval completed, whatever the verdict (a regressed result is still a result), 2 usage, 3 it could not run. --keep keeps the temp directory, with the platform’s own result and its HTML report.

bob eval gate (scripts/evals/gate.js)

bob eval gate commit /tmp/commit.json
bob eval gate commit /tmp/commit.json --record --pr '#4300' --pages commit-skipped-hook

The rule: pass only when all three hold (#4259, the proposer contract in skill-evolution):

with is strictly higher than the best recorded for the same model. If no best is recorded for that model yet, the floor is the result’s own without score, so the first baseline must at least beat running with no item loaded.
with is strictly higher than without: the change must help at all, even when it beats the recorded best. With no best recorded, this is the same check as the first.
trigger_rate is not below the recorded best’s trigger_rate for the model. When either side is null, or no best is recorded yet, this check is skipped and the reason says so.

An equal score is a reject, and a partial run is never accepted. A reject’s reason names every condition that failed. Exit codes: 0 accept · 1 reject · 2 usage.

--record writes the outcome:

On accept, it raises the model’s best in baseline.json, writing score, without, trigger_rate, run and date. A reject leaves the baseline as it was.
Accepted or rejected, it appends one row to wiki/skill-impact.md (Date · Skill · PR · Pages cited · Score before · Score after · Outcome). The row needs --pr and --pages, which name the proposal’s pull request and the existing wiki/patterns/ pages it was built on, so every row passes make check-wiki.

baseline.json:

{
  "schema": "skill-eval-baseline/1",
  "item": "commit",
  "best": {
    "claude-sonnet-5-5": { "score": 0.83, "without": 0.67, "trigger_rate": 1, "run": "eval-commit-20261009T084444Z", "date": "2026-10-09" }
  }
}

A new suite starts with "best": {}. An entry recorded before #4259 has no trigger_rate; it reads as null, so the trigger check is skipped against it until the next accepted change records one. No migration is needed.

make evals

EVALS=1 make evals                       # every suite, --runs 1, $0.50 ceiling each
EVALS=1 EVALS_MODEL=haiku EVALS_MAX_COST_USD=0.2 make evals

Without EVALS=1 the target prints that it skipped and exits 0. It is not on the tests/test-* glob and not in make ci, so the farm runner never pays for it until the cost is understood. It fails only when a suite could not run. A no-signal or regressed verdict is reported, not failed.

The free test is tests/test-skill-evals.js, which is in make test. It drives bob eval run, bob eval gate and bob u exam end to end with only claude stubbed.

Suites today

Suite	Cases	What `fires` scores
`commit`	`fires`, `quiet`	A `fix` subject under 72 characters that says why, with no AI boilerplate
`tdd-flow`	`fires`, `quiet`	A failing test shown before any implementation, said to be run red first

The prototype area’s suite is authored by #2392, which picks that area from evidence.

First runs (2026-10-09, --runs 1): on haiku and on sonnet both suites scored the same on both arms (no-signal), and neither skill fired on its fires case, although both stayed quiet correctly. The trigger graders exist to surface this gap. The suites were not tuned to hide it.

Known limitations

The strict gate rejects neutral changes. A change that scores the same as the best is rejected, even when it would pay off later, for example a clearer structure that a following edit builds on. The paper behind #2387 notes the same cost. It is accepted here and the gate is not weakened. The wiki keeps what a rejected change taught.
Few runs are noisy. --runs 1 is a smoke test. Record a baseline from the default of 3 runs or more.
The no-item arm is not a clean room. The eval child is a normal claude run for the operator. Whether it also loads the operator’s own skills from BOB_HOME depends on the platform, and this is not confirmed. A globally deployed skill may therefore be available on both arms. That would shrink the delta, but it does not affect the trigger graders, which are pinned to the item by input_match.