Skip to content

Skill evolution: traces, a wiki and skills

Skill evolution: traces, a wiki and skills

Section titled “Skill evolution: traces, a wiki and skills”

BoB records a great deal of experience. It keeps session ledgers, decision logs, harness defects, loop notes, memory files, lesson issues and chunk reports. None of it changes a skill. /trace-mining edits memory and CLAUDE.md and files issues. Loop notes become bugs. Skill edits are made by hand, with no record of the evidence behind them, and nothing measures whether an edit helped.

The skill-evolution capability (#2387) closes that loop. It takes its shape from one paper, WikiSkill. This page is the design the capability’s requirements implement: what the paper found, where each BoB surface sits in its model, where the wiki lives, and what the maintainer and the proposer may read and write. The wiki store (#2388), the maintainer (#2389), the eval gate (#2390), the proposer (#2391) and the prototype (#2392) build against it.

Tang, Rashtchian, Ferng, Tomkins, Juan and Vu, WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution, Google Research, arXiv:2608.27454v1, 27 August 2026. Every figure below was read from the PDF (28 pages), not from an HTML summary. Section, table and appendix references are the paper’s.

The workspace has three layers, and each has its own rule for change (§3.1, Figure 2).

Layer Holds Rule
Raw (raw/) Every execution trace from the training rollouts: reasoning, tool calls, tool outputs, final answers. Immutable. Written once.
Wiki (wiki/) patterns/ (one markdown page per failure mode or successful strategy, with workarounds), index.md (the catalog), logs.md (the evolution log), skill-impact.md (one entry per proposal). Compounds. Never reset, never rolled back.
Skills (skills/) The active skill set. Each skill is a SKILL.md and a PURPOSE.md that maps the skill back to the wiki patterns that motivated it. Reversible. A change is kept only if validation improves.

Each iteration has four steps (§3.2, Algorithm 1).

  1. Inference agent. It runs the training tasks with the active skills injected in full into its system prompt. It cannot read the wiki.
  2. Wiki maintainer. It gets the whole wiki and a stratified sample of the new traces: at most 8 per iteration, at most 5 failing and 3 passing, each trace cut to 15,000 characters (Appendix C). It does root-cause analysis on the failures and pulls strategies from the passes. It creates or patches pattern pages (append, replace, insert — never a rewrite), revises index.md and appends one entry to logs.md. It is one LLM call per iteration. There is no cap on how many pages it touches.
  3. Skill proposer. A multi-turn ReAct agent, about 10 to 20 turns (Appendix D.2). It starts with only the wiki index, skill-impact.md and a short summary of every training outcome (pass or fail, prediction, ground truth). It opens pattern pages and raw traces on demand. It emits one atomic proposal that targets one skill: a new skill or one patch to an existing one.
  4. Gate and rollback. The candidate skill set runs on the validation split. It is accepted only if its score is strictly higher than the best so far (equation 4). The best score starts at the empty skill set’s validation score, and the loop stops early if validation reaches 1.0. A rejected candidate is discarded and the skills revert. The wiki is not rolled back either way. After every gate, the harness — not a model — appends the proposal, the target skill, the unified diff, the validation score and the outcome to skill-impact.md. That ledger is how the proposer avoids proposing a rejected change twice.

Five benchmarks (LiveMath, SealQA, SpreadsheetBench, OfficeQA, ALFWorld) and five models (Qwen-3.5-4B, Qwen-3.5-9B, Qwen-3.6-27B, Gemma-4-31B, Gemini-3.5-Flash). Each method ran three independent evolutions, and scores are the mean test result. Significance is a paired bootstrap with 1,000 iterations at p < 0.05. Evolution ran for up to 8 iterations (0–7, Table 5). The splits are small: 16 to 80 training tasks and 10 to 40 validation tasks per benchmark (Table 6).

  • Against other methods (Table 1). WikiSkill had the best average for all five models. Its lead over the strongest other method was 3.3, 5.1, 10.0, 5.8 and 12.0 points (Qwen-4B, Qwen-9B, Qwen-27B, Gemma-31B, Gemini-Flash). It was not best on every cell: Qwen-4B on OfficeQA fell below its no-skill score (30.2 → 28.5).
  • Model scale (§4.2.1). The gain over no skills grew with Qwen size: +12.3, +17.5 and +23.9 points. Qwen-3.5-9B with evolved skills (47.4%) beat Qwen-3.6-27B without them (39.4%).
  • Transfer (Table 2). Skills evolved by one model often helped another, and sometimes more than the target’s own skills. On ALFWorld, Qwen-3.5-9B reached 70.2% with Qwen-3.6-27B’s skills against 63.4% with its own. Transfer can also hurt: Qwen-3.5-4B’s SpreadsheetBench skills dropped Gemini-3.5-Flash from 50.5% to 18.1%, while Qwen-3.6-27B’s raised it to 63.4%. The paper traces the loss to low-level workarounds (one-line Python commands, string-conversion rules) that help a small model and constrain a strong one, and to fragmented procedures whose extra tool calls exhausted the stronger model’s budget.

Table 3 varies wiki access for the inference agent and the proposer, on Gemini-3.5-Flash, across four benchmarks (ALFWorld is excluded because that model scores 100% on validation before any evolution). When the proposer has no wiki access, the maintainer is removed too, so nothing accumulates.

Inference agent reads wiki Proposer reads wiki Average
— (no skills) — 40.4%
yes no 45.3%
no no 48.7%
yes yes 60.9%
no yes (the default) 63.7%

Two findings follow.

  • The wiki carries most of the gain: 48.7% → 63.7% (+15.0 points) when the proposer gets a wiki.
  • Giving the running agent the wiki makes the skills worse: 63.7% → 60.9%, and LiveMath falls from 72.6% to 64.8%. The paper’s hypothesis is that an agent that can read the wiki solves tasks from it rather than from the skills, so its traces say less about what the skills lack.

Correction. The capability #2387 quoted the ablation as “63.7% vs 42.0% without it”, from a summarising fetch of the HTML. The PDF’s figure is 63.7% vs 48.7%. 42.0 is a single cell (SealQA, inference-agent-only row), not an average. The 63.7% → 60.9% figure for inference-agent access is correct.

  1. No triggering. Skills are injected in full, to remove retrieval and triggering as variables. The paper does not measure whether an agent picks the right skill. That is BoB’s main weakness (the ledger’s no-trigger records), so BoB’s eval suites must measure it.
  2. A strict gate. Only a strictly better score is accepted. A neutral change that might enable a later gain is lost.
  3. No pruning. The wiki only grows. Table 4 notes that every pattern creation and edit is kept. The paper names pruning as future work.
  4. No very long tasks. Nothing spans hundreds of actions or hours. A warp-drive run does both.

Two further points matter for BoB. The validation splits are small (10 to 40 tasks), so one task can flip a gate; the paper offsets this with three runs and a bootstrap, and a single BoB run will not have that. And every result is on benchmarks with ground-truth answers, while most BoB work has no ground truth, only checks.

BoB already has most of the three artifacts. What it lacks is the middle one, and the line between them. The table places every surface that records experience or changes a skill. “Lacks” is what keeps it from playing its part in the loop.

Surface Writes it Reads it Lacks
Session ledger — .claude/session-ledger.jsonl (reference) hooks/session-ledger-append.sh (PostToolUse on Skill, Task, Agent); backfill-transcripts.js; gap-analysis.js observe for improvised and no-trigger gap-analysis.js, gap-detect.js, fleet-usage.js, bob tour, /trace-mining §5b Only invocations, not outcomes. A no-trigger record is the one trace that measures the paper’s blind spot, triggering, and nothing consolidates it.
Decision log — .claude/.decisions.jsonl (scripts/decision-log/README.md) log-decision.sh, fed by the capture-decision.sh hook and log-code-review.sh /trace-mining §1b, distill.sh Gitignored and per project. Its only consolidation target is memory and CLAUDE.md, never a skill.
Harness defects — harness_defects[] in .claude/.warp-drive-state.json (state reference) recordHarnessDefect() in the state machine, when an agent hand-fires a hook’s event harness-defect-report.js (files a harness-defect bug per fingerprint, adds a loop note), harness-defect-summary.js (dashboards, brief, audit) Dies with the state file. Becomes a bug, not knowledge about the skill or hook that failed.
Loop notes — .claude/loop-notes.json (dev-lifecycle) loop-notes.js add from the agent; harness-defect-report.js; promote writes back the defect tag hooks/loop-notes-inject.sh (SessionStart), run-checks.js (flaky quarantine), bob tick’s digest, rituals Closest thing BoB has to a wiki page: signature, count, last_seen, defect-tag retirement. But it is per project, capped at 50, and injected into the running agent — the configuration the ablation found worse.
Lesson issues — label lesson (/journal) /journal lesson, lessons-extractor, bob do, /retrospective /review-lessons, gap-analysis.js --lessons-json, trace-mine-session.sh, the dashboard Read only when someone queries the label. Nothing surfaces a relevant lesson at work start (#106’s remaining gap).
Chunk and session reports — chunk-report, session-summary issues The agent, from the state machine’s phase text; trace-mine-session.sh files a session-friction issue /trace-mining §1 (last five summaries), dashboards, check-report-labels.js Narrative only. The iteration trail is the nearest thing to a failure trace, and it is read by a human or not at all.
Tick digests — $BOB_HOME/fleet/ticks/<date>.json (tick-digest/1) scripts/tick/tick.sh Dashboards, bob brief, project-live.js Counts, not causes. It records loop_notes_added, not what the notes say.

Knowledge (the wiki layer, which BoB does not yet have)

Section titled “Knowledge (the wiki layer, which BoB does not yet have)”
Surface Writes it Reads it Lacks
Memory files — <config>/projects/<path>/memory/*.md Claude Code’s auto-memory; distill-memory.sh piped into self-update.sh The harness at session start; /memory-audit; /memory-recall Per machine and per project, read by the running agent, never by a skill author. Has the retirement rules the wiki needs: defect: tags, ## History, staleness.
/trace-mining (registry/commands/trace-mining.md) — Session summaries, the decision log, the ledger It is the nearest thing to a maintainer. Its outputs are a report issue, authoring-gap issues, and memory and CLAUDE.md edits. None of them is a persistent, cited, consolidated page, and none is a skill change.
distill.sh (#332) — The decision-log count A trigger, not a consolidator: it fires /trace-mining every 75 decisions and rotates the log.
self-update.sh — .claude/.self-updates.jsonl Itself, on every applied edit self-update.sh log, at session end The model for skill-edit provenance: target, before and after SHA, lines changed, summary. But it guards memory and CLAUDE.md, not skills, and records no score.
lessons-extractor (registry agent) lesson issues — Proposes docs, not skills. Its disposition is needs-work.
Surface Writes it Reads it Lacks
Skills, commands and agents — skills/, commands/, agents/, registry/ Hand edits; skill-creator (the target of authoring-gap issues) The running agent, by trigger or provisioning No PURPOSE.md: nothing links a skill back to the evidence for it. No score before or after an edit.
bob promote (scripts/cli/cdp) Copies a project’s local item into BOB_SOURCE and links it back — Moves a skill between scopes. The SkillSpector scan is a safety gate, not a quality gate.
bob tour — .claude/.tour.json (self-discovery) --tried, --park The owner; the provisioning nudge Measures whether an item is used, not whether it is good.
bob u dispositions — registry/dispositions.json (reference) bob u dispose --apply check-dispositions.js, bob u The only surface that already reads an eval result: evidence.exam holds a skill-eval/1 document, and certified cites a suite’s baseline.json. The runner it calls, scripts/evals/run.js, does not exist yet; #2390 must build it to that shipped contract.

Three gaps follow. Traces are scattered across projects and gitignored. Nothing consolidates them into knowledge a skill author reads. And nothing scores a skill change. The capability adds the middle layer and the score, and reuses the rest.

Decision (2026-09-14): the wiki is a git-tracked wiki/ directory at the root of BOB_SOURCE. It holds knowledge about BoB’s tooling and skills only. A fact about a client’s code stays in that client’s repo.

Why there: the skills it improves live in BOB_SOURCE, so a page and the skill change it motivated land in the same repository and the same PR history. Git gives the “never rolled back” rule for free: the wiki only moves forward through commits, and a rejected skill PR leaves the wiki commit that informed it in place. And it keeps the wiki out of BOB_HOME, so it is never deployed into a running session.

wiki/
├── patterns/<slug>.md authored — one page per failure mode, strategy or workaround
├── index.md derived — generated from the pages' frontmatter
├── logs.md appended — one entry per maintainer run
└── skill-impact.md appended — one row per proposal, written by the gate
File Kind Written by Rule
patterns/<slug>.md Authored The maintainer, through a PR a human merges One page per pattern. Frontmatter: kind (failure-mode, strategy or workaround), skills[] it bears on, evidence[] (issue numbers, ledger record ids, loop-note signatures, session ids — references, never copied trace text), first_seen, last_seen, count, optional defect: owner/repo#n, status (active or retired). Edits are patches: merge evidence, bump count, extend last_seen. A page is never deleted.
index.md Derived scripts/wiki/index.js Regenerated from page frontmatter; make check fails when it is stale. The paper has the maintainer write the index; BoB derives it, so it cannot drift from the pages.
logs.md Appended The maintainer run One entry per run: date, sources, records read, records refused by the scrub, pages created and updated. Never rewritten.
skill-impact.md Appended The gate harness, not a model One row per proposal: skill, PR, pages cited, score before and after, with and without the skill, accepted or rejected. As in the paper, the harness writes it so the audit trail is not a model’s account of itself.

workaround is a separate kind from failure-mode and strategy because of the transfer result: low-level workarounds helped the model that found them and hurt a stronger one. A proposer treats a workaround page as evidence about one model and one harness, not as a procedure to put in a skill.

The paper’s wiki only grows, and it calls pruning an open problem. BoB already prunes its other knowledge stores, and the wiki uses the same three rules.

  1. Defect closed. A page tagged defect: retires when that issue closes, checked through scripts/lib/defect-status.js — the rule loop notes and memories already follow (#1837). A workaround for a fixed bug is not knowledge any more.
  2. Recurrence. A repeat is evidence on the existing page, not a new page: the maintainer matches by signature (loop-notes’ signature()), bumps count and extends last_seen. A pattern that recurs after its page retired reopens that page, with the recurrence recorded in logs.md.
  3. Staleness. A page whose last_seen is older than a configured age is reported by scripts/wiki/retire.js, as /memory-audit reports stale memories. A report, not a deletion: an old pattern may still be true.

A retired page stays in git and leaves the index, so the proposer stops seeing it while the history stays readable.

The page format, the lint, the table row formats and the retire command are specified in the Wiki Store reference.

The maintainer turns traces into wiki pages (#2389).

  • Inputs. The current wiki, and a trace packet: every record added since the last run to the trace stores in the first table above — loop notes, the decision log, harness defects, ledger no-trigger and improvised records, lesson issues and session reports. Each record carries a reference a page can cite. A watermark records the last position read in each source.
  • Output. Patch edits to wiki/patterns/: create a page, or update one (merge evidence, bump count, extend last_seen, refine the text). Never delete and never rewrite. Then one entry in logs.md. A run opens one PR. The index is regenerated, not written.
  • Bounds. The paper caps what the maintainer sees (8 traces, 15,000 characters each). BoB caps the packet the same way, so one run is one bounded agent call, and records it leaves out wait for the next run.
  • Idempotence. A second run over the same inputs changes nothing. The watermark makes that true even when a run’s PR is still open.
  • Where it runs. From /trace-mining and bob tick, never as a warp-drive phase: the state machine is frozen until 2026-10-11 (#2118), and consolidation belongs between runs, not inside one. It is bob wiki maintain; the run contract is in the wiki reference.

The proposer turns wiki knowledge into one skill change (#2391).

  • Inputs. Three documents, as in the paper: wiki/index.md, wiki/skill-impact.md and the eval outcome summary for the target area (the skill-eval/1 document #2390’s runner writes). From there it may open pattern pages and the traces they cite, on demand.
  • Not an input: the raw optimization history. It does not get earlier proposers’ transcripts, a dump of the decision log or the ledger, or a flat list of past attempts. What it needs from the past is already compiled: the reasons in the pages and the outcomes in skill-impact.md. That compilation is the paper’s whole point. EvoSkill’s flat proposal history is the design WikiSkill beat.
  • Output. One atomic change to one item under skills/, registry/skills/, commands/, registry/commands/, agents/ or registry/agents/: a new item, or one patch to an existing one. Never two items, and never a wiki edit.
  • Provenance. The change opens as a PR labelled skill-proposal. Its body cites the pages it built on and the traces behind them, and shows the gate’s document: the score with and without the item, before and after the change, the trigger rate, the model and the cost. The paper keeps a PURPOSE.md beside each skill. BoB keeps that link in the PR and in the skill-impact.md row instead, because commands and agents are single files and a second file would be deployed into BOB_HOME with them.
  • The gate. Accepted only when the changed item scores strictly higher than its recorded baseline, scores higher with the item than without it, and does not trigger less often. The last two go beyond the paper: it injected every skill, so it never had to ask whether a skill helps at all or whether it fires. A rejected proposal still opens, as a draft, and its rejected row in skill-impact.md is what stops the next proposer from trying it again.
  • A human merges. The proposer opens PRs and never merges one. This keeps the gap analysis’s “zero auto-authoring” stance. An auto-merge on a strict eval win, like the provisioning fast path (#1478), would need its own decision issue first.

The contract above is built as bob wiki propose (scripts/wiki/propose.js) and the agent prompt registry/agents/skill-proposer.md. One run is:

  1. Pick the target. Either the item named, or with --auto the item with the most new evidence. New evidence is the evidence refs on active pages that name the item, counted on pages seen since the item’s last skill-impact.md row. Only an item with an eval suite (evals/<item>/) can be picked, because the gate needs a score. An item with evidence but no suite is listed, so the next suite to author is plain. An open skill-proposal PR for the item ends the run as a no-op.
  2. Ask the agent. One headless claude -p call, in a worktree of BOB_SOURCE on a skill-proposal/<item>/… branch, with Read as its only tool. It gets the item’s file, the wiki index, the ledger, the earlier proposals for the item (titles and states only) and the item’s skill-eval/1 summary: the best score per model and the case names. It answers the whole new text of the item’s one file and the pages it built on. The script refuses any other path (so wiki/, CLAUDE.md and docs/vision.md are out of reach), a page that is not about the item, and a tree the agent wrote to. Today the change is a patch to an existing item; a new item stays an authoring-gap issue.
  3. Gate it. bob eval run on the changed item, then bob eval gate. The gate enforces all three conditions of the contract above: strictly higher than the best recorded for the model (or than the no-item arm when nothing is recorded yet, the paper’s rule, #2390), higher with the item than without it, and a trigger rate no lower than the recorded best’s (#4259). A rejection names every condition that failed.
  4. Open the PR. The change is committed and pushed, then opened as one PR labelled skill-proposal, as a draft when the gate rejects it. Its body holds the diff, the pages cited, their traces, the score with and without the item, the delta, the trigger rate, the model, the cost and the gate document.
  5. Record it. bob eval gate --record appends the skill-impact.md row on the same branch: the date, the item, the PR, the pages, the score before and after, accepted or rejected. An accepted change also raises evals/<item>/baseline.json. Nothing in the wiki is rolled back. A rejected draft that is closed unmerged takes its row with it, but its PR is still among the earlier proposals the next run is shown.

/trace-mining offers the proposer as its last step. bob tick runs it only when the tick’s project is bigbrain: for real at automation Level 3 or 4, as a dry run below that. See Consolidate Traces into the Wiki.

The wiki is written by the maintainer and read by the proposer. Nothing else reads it: no SessionStart hook, no provisioned item, no command a running agent loads. The runtime surface stays the skills.

The evidence is the ablation. Giving the running agent the wiki as well lowered the result from 63.7% to 60.9%, and LiveMath from 72.6% to 64.8%. The paper’s explanation is that an agent with the wiki solves tasks from it, so its traces stop showing what the skills are missing. In BoB the running agent is also the trace source — every warp-drive run feeds the next maintainer run — so the same effect would blind the loop to the gaps it exists to close. #2388 adds a check that no hook, command or manifest references wiki/.

BoB already breaks this rule once. hooks/loop-notes-inject.sh injects each project’s loop notes, which are wiki-like knowledge, into every session at start (#588). Whether that helps or hurts is an open question, and the design does not settle it by argument. The prototype carries it as an acceptance criterion: #2392 runs its area’s eval suite with the injection on and off, records both scores, and files a decision issue to keep the injection, restrict loop notes to the proposer, or retire the hook. The measurement must look at two things, because the paper’s harm was to the traces, not to the run: the task score, and whether the run’s traces still show the failures the notes paper over.

The wiki is in the product repository, and every client installs from a tag of it. So a client project’s traces may teach the wiki about a BoB item, but no client fact may reach it.

Before the maintainer. A trace packet from any project other than bigbrain is scrubbed before the maintainer sees it. Each record is checked against the operator profile’s tenants[] literals (repo, org, domain and product names) and the credential patterns in registry/skills/secrets-management/references/secret-patterns.json — the same rules as the release gate. A record that matches is refused, never redacted and passed on, and the count of refused records goes into logs.md. With no operator resolved there are no tenant literals to check against, so every record from a project other than bigbrain is refused: the scrub fails closed, as the release gate does.

What a page may carry from a client trace. Generalised knowledge about a BoB item — “the docs hook does not fire after a gh issue edit” — never the client’s code, file paths, issue text or names. Its evidence refs are opaque: the trace kind and a hash of the record (ext:<kind>:<hash>), which the maintainer can resolve on the machine that holds the client’s store. A client’s repo name, issue URL or path never appears in a ref.

Two gates behind it. The wiki lint (#2388) runs the tenant and credential rules over wiki/ in make check, so a leak fails on the PR that adds it. The release content gate (#2290, reference) already covers wiki/ without a change: its tenant and credential rules read every tracked file. What must stay true is that scripts/checks/release-content-allow.json never lists a wiki/ path.

BoB already has an outer improvement loop. The capability’s rule is that it ends up with one, so each existing piece either becomes a step of the new loop or keeps a job the new loop does not do.

Piece Today In the loop
/trace-mining Reads session summaries, the decision log and the ledger; files a report issue and authoring-gap issues; edits memory and CLAUDE.md through self-update.sh. The maintainer’s entry point. It runs the maintainer over the trace packet, and its report links the run’s logs.md entry. An authoring-gap finding — a skill BoB lacks — becomes evidence on a page, and creating that skill becomes the proposer’s job, gated like any other change. Its memory and CLAUDE.md edits stay: they record the operator’s preferences and the project’s rules, which belong to the running agent, not to a skill.
Decision distillation (#332: distill.sh, self-update.sh) Counts decisions, fires /trace-mining every 75, rotates the log; guards and records every instruction edit. The trigger and the provenance model. distill.sh stays the count trigger, so a busy project feeds the maintainer without a schedule. The decision log becomes one of the maintainer’s sources. .self-updates.jsonl is the record skill-impact.md extends, adding the evidence cited and the scores. self-update.sh keeps guarding memory and CLAUDE.md. A skill change never goes through it: it goes through a PR and the gate.
#106’s remaining gap — surface a relevant lesson when work starts Lessons are read only by label query. Answered by the loop, not by injection. Showing a lesson to the running agent at work start is the configuration the ablation found harmful. Instead, lesson issues become maintainer input, and the lesson reaches the next run as a better skill. #106 is closed, and nothing should build lesson injection before #2392’s loop-note measurement says whether start-of-session knowledge helps.
SkillClaw evaluation (#1240) A spike: adopt SkillClaw, borrow from it, or neither. Settled by comparison with this design: a pass (closed 2026-10-01, the evaluation). It failed on all five points below, so BoB runs one evolution loop: this one.

SkillClaw records sessions through a local API proxy and evolves a skill library from them, optionally on a shared server. #1240 decides it by putting it next to this design on these points, one line each:

  1. Three artifacts. Does it keep a persistent knowledge layer apart from its session records and its skills, or does it go from sessions straight to skill edits — the design the WikiSkill ablation beat (48.7% against 63.7%)?
  2. The gate. Does it accept a change only on a measured improvement, and can it run against claude plugin eval suites and the skill-eval/1 contract?
  3. Who writes skills. It defaults to writing ~/.claude/skills directly. BoB’s skills live in BOB_SOURCE and reach BOB_HOME only through deploy.sh, and a human merges every change. Can it run in a proposal-only mode that opens a PR?
  4. Client data. Its proxy sees every request, and its server reads shared storage. Can it run with the client-data boundary above — nothing from a client project leaving the machine unscrubbed?
  5. What it does that this design lacks. Deduplication across the library is the obvious candidate. A part that wins on its own goes into the loop as a component; the tool does not replace the loop.