SkillClaw Evaluation

SkillClaw Evaluation (#1240)

Evaluated 2026-10-01 by reading the source through the GitHub API at commit main of 2026-08-17. Nothing was installed or run; why is under What was not run.

Verdict: pass. SkillClaw is not adopted, and none of its parts go into the #2387 skill-evolution loop. Its Claude Code integration rewrites the deployed ~/.claude/settings.json and sends every Claude Code request, translated to the OpenAI chat format, to a different model provider. It evolves skills straight from sessions, judges them with a model rather than an eval, and writes them into the deployed ~/.claude/skills. Each of those breaks a rule #2387 or BoB’s deploy model already sets. The one part #2387 asked about, deduplication, turned out to be narrower than described: it merges two versions of the same skill name, not overlapping skills across a library.

The source

Claim in the issue	Finding
Repository	AMAP-ML/SkillClaw, the only match with real activity (the four other skillclaw repositories have 0–5 stars).
Licence	MIT.
Maturity	Created 2026-04-10, about 2,650 stars, 40 open issues, no tagged releases. Last push 2026-08-17: a batch of fixes (unsafe skill-bundle paths, a bounded truncation loop, blocking storage I/O). Python, with a test suite and a Ruff lint workflow; no test workflow in CI.
Paper	arXiv 2604.08377, “SkillClaw: Let Skills Evolve Collectively with Agentic Evolver”, submitted 2026-04-09. Confirmed through the arXiv API.
Claude Code integration	Added 2026-04-20, with doctor and restore commands.

How it attaches to Claude Code

skillclaw/claw_adapter.py (_configure_claude) does three things when the proxy starts with the Claude Code integration:

1. Creates ~/.claude/skills if missing and copies in any skills from its own legacy folder.
2. Loads ~/.claude/settings.json, sets env.ANTHROPIC_BASE_URL to http://127.0.0.1:<port> and env.ANTHROPIC_AUTH_TOKEN to its proxy key, and writes the whole file back. A timestamped copy of the old file is kept in ~/.skillclaw/backups/claude/.
3. From then on, every Claude Code request goes to the proxy.

The proxy’s /v1/messages handler (skillclaw/api_server.py) converts the Anthropic request to an OpenAI chat body, can rewrite a new session’s opening prompt, injects retrieved skills into the system message (up to 30,000 characters by default), records the turn, and forwards it to the configured provider: any OpenAI-compatible endpoint, OpenRouter or AWS Bedrock. Claude Code has no session header SkillClaw understands, so session boundaries are guessed from request patterns (doctor says so).

restore claude writes the newest backup over settings.json, whole.

Against BoB:

• Model access. Claude Code would no longer talk to Anthropic directly. It would run on whatever provider SkillClaw is pointed at, paid by that provider’s key, through a translation to and from the OpenAI format. This alone rules it out for BoB’s machines.
• settings.json ownership. The file is machine-specific and protected, and its .hooks block is reconciled by deploy.sh (#1918). BoB refuses to rewrite it while any session is live, because that silences the hooks of every running session (#2040). SkillClaw rewrites it on proxy start with no such check.
• Restore can lose changes. Restoring the latest backup undoes everything written to settings.json since, including a hook reconcile by a deploy that ran in between.
• Runtime injection. Skills pasted into the system prompt by similarity bypass Claude Code’s own skill triggering, and put knowledge in front of the running agent, which #2387 avoids on the WikiSkill ablation’s evidence.
• Client data. The proxy sees every request from every project, including client projects, and its optional evolve server reads them from shared storage (Alibaba OSS, S3 or a local folder).

The evolution loop

The evolve server’s workflow engine runs:

1. Summarise each recorded session with a model.
2. Judge sessions without a score with a model (session_judge.py).
3. Aggregate sessions by the skill they used.
4. Execute: create a skill, evolve an existing one, or merge two versions with the same name (execute_merge, a model prompt to “combine the best parts of both”).
5. Verify (optional): a model reviews the candidate and accepts or rejects it (skill_verifier.py).
6. Publish to shared storage, from which every client pulls.

The agent engine replaces steps 4–5 with an agent (OpenClaw) editing skill files directly.

Validated publish

With EVOLVE_PUBLISH_MODE=validated, a candidate is staged as a validation job instead of being published. A client that has opted in picks the job up when it has been idle for a while (the README’s example: 300 s idle, at most 5 jobs a day), replays it and scores it with a process reward model, which is another model prompt (validation_worker.py, prm_scorer.py). A later evolve cycle publishes the candidate once enough scores pass the threshold.

As a quality gate for BoB this is weaker than what #2387 already plans (#2390): every judgement is a model’s opinion, nothing compares the skill’s score before and after the change, nothing checks with and without the skill loaded, and nothing checks whether the skill triggers. A human never sees the change before every client receives it.

SKILL.md compatibility

Field	SkillClaw	BoB registry
name, description	Required; a skill without them is skipped	Required
category	Optional; written as general when absent	Orchestrator category (ai-tooling, …)
metadata	Optional	Not used
applies_to, recommended_for, default, required_with	Kept as “extra frontmatter” and written back in YAML flow style	Read by recommend.js and the provisioning engine
Bundled files (references/, scripts/)	Supported as a skill bundle, with path-safety checks since #81	Common

The format reads BoB’s skills without loss. Writing them back is the risk: the evolve and merge prompts tell the model to keep extra frontmatter only “when genuinely useful”, so an evolved BoB skill can come back without the orchestrator fields that decide where it is provisioned. A merge across versions does the same.

Against the #2387 design

The comparison skill-evolution.md asks for, one line each:

1. Three artifacts. No. Sessions go straight to skill edits; there is no persistent knowledge layer between them. That is the arrangement the WikiSkill ablation beat (48.7% against 63.7%).
2. The gate. No measured improvement is required. Acceptance is a model’s verdict, optionally followed by model-scored replays. It cannot run claude plugin eval suites or produce a skill-eval/1 result without being rewritten.
3. Who writes skills. It writes the skills folder the agent reads, which for Claude Code is the deployed ~/.claude/skills, and every client pulls published changes automatically. There is no mode that opens a pull request against a source repo.
4. Client data. The proxy records every request from every project, and shared storage moves the records off the machine. There is no scrub step.
5. What it does that the design lacks. Deduplication is a same-name merge of two versions, done by a model prompt. It does not find two differently-named skills that overlap. The skill-usage counter (record_feedback: positive and negative counts per injected skill) is covered by #2387’s skill-impact ledger and the session ledger’s no-trigger records, which BoB measures without a proxy.

Nothing passes point 5, so nothing is filed. If overlap between skills becomes a problem, it belongs in #2387’s proposer as a proposal kind (merge two skills), gated by the same evals.

Compared with /trace-mining and #332

BoB’s existing loop reads its own records (session summaries, the decision log, the session ledger) after the fact and edits memory and CLAUDE.md through self-update.sh, which guards and logs each edit. #2387 turns it into the entry point of one skill-evolution loop. SkillClaw would be a second loop with its own capture (a proxy), its own store and its own publisher, which is what #2387’s “one loop, not two” rule exists to prevent.

What was not run

The issue’s install, live-session and evolve-server criteria were not run, on purpose:

• The Claude Code integration sends every request to a non-Anthropic provider, so a real session needs an OpenAI-compatible, OpenRouter or Bedrock key and sends the session’s content there.
• Starting it rewrites ~/.claude/settings.json on a machine with live sessions, which BoB’s own deploy refuses (#2040).
• The pass verdict comes from the architecture, not from behaviour a trial would show. A trial could only confirm it.

Should a trial be wanted anyway, it belongs on a machine with no live BoB sessions, with HOME pointed at a throwaway directory so that ~/.claude is a copy, and with a provider key bought for the purpose.

Also not verified: the paper’s results, and the agent engine’s behaviour.