Agent-Tool Evaluation 2026-07

Trending Agent-Tool Evaluation — July 2026 (#544)

Evaluated 2026-07-17. Source: a weekly roundup of fast-rising GitHub AI repos (tip source) surfaced seven tools relevant to BoB’s territory (skills, MCP, orchestration, token economics). Each was researched against its live repo, docs, and independent commentary; verdicts weigh BoB’s specifics — a Claude Code tooling framework running unattended L3 loops on hosted Anthropic APIs, with its own ~60-item skill registry, per-project provisioning, and JSONL token accounting.

Verdict summary

Tool	What it is	Verdict
headroom	Pre-LLM context compression (lib/proxy/MCP)	Pass for now — re-evaluate as a bounded A/B in ~2 quarters
agent-skills (addyosmani)	24 curated SKILL.md workflow skills	Pass on adoption; harvest patterns
SkillSpector (NVIDIA)	Static security scanner for agent skills	Adopt (wrapped) — static-only CI gate + committed baseline
codebase-memory-mcp	Code knowledge-graph MCP server	Pass for now — revisit at scale
agentsview	Local analytics over agent session history	Wrap — cautious read-only complement, pending trial
LMCache	KV-cache layer for self-hosted inference	Pass — categorically inapplicable
flue (withastro)	TypeScript autonomous-agent harness	Pass on adoption; mine as reference

Fit assessments

headroom — pass for now

headroomlabs-ai/headroom (Apache-2.0, ~60k stars but only ~6 months old, effective bus factor of 1). Compresses agent-bound context via content-type-routed algorithms: statistical JSON compression (the 60–95% headline), AST-aware code compression (a claimed 15–20%), an ML model for prose, with originals cached locally and retrievable on demand. A Claude Code insertion point genuinely exists — headroom wrap claude points ANTHROPIC_BASE_URL at a local proxy (verified in source) — but the issue tracker shows routine breakage against Claude Code releases (silent no-op compression, 30s hangs on long conversations, remote-control breakage). A single-maintainer MITM proxy over all Anthropic traffic — auth tokens and content included — is the highest-trust position in the stack, and its failure modes are exactly the ones that corrupt an unattended warp-drive run. Their own README now says ~20% for coding agents, not 60–95%. See the token-savings test below: BoB can capture most of the realistic gain natively. Re-evaluate in ~2 quarters as a bounded experiment (one cdfork worktree, API-key session, measured against a control) if token cost becomes a pressing concern.

agent-skills — pass on adoption; harvest patterns

addyosmani/agent-skills (MIT, ~79k stars, active): 24 SKILL.md workflow skills organized by SDLC phase, plus personas, checklists, and slash commands. Format-compatible with BoB (standard SKILL.md frontmatter, near-identical directory layout) but a direct functional overlap with BoB’s existing library — same lifecycle slots, minus BoB’s provisioning manifests, orchestrator metadata, and CI validation. Wholesale adoption would create ~24 trigger-colliding, metadata-poor registry items whose instructions conflict with BoB conventions (git workflow, attribution, issue tracking). Where it beats BoB’s median skill is technique: anti-rationalization tables, mandatory verification exit criteria, red-flag lists, the “prove-it” bug-fix pattern. The author explicitly blesses “read as specification” use. Harvest via a one-time comparative audit of BoB’s overlapping skills against their counterparts — a grooming task per skill cluster, not a dependency.

SkillSpector — adopt, wrapped, as a static-only CI gate

See the dedicated evaluation below. Short version: input contract matches BoB registry items exactly, CI ergonomics (exit codes, SARIF, committed baseline) are Prime-Directive clean, and its threat model (26% of wild skills vulnerable, per the NVIDIA dataset) bites hardest exactly where BoB is most exposed — third-party items entering the registry.

codebase-memory-mcp — pass for now, revisit at scale

DeusData/codebase-memory-mcp (MIT, ~32k stars in ~2 months — treat virality skeptically; single pseudonymous maintainer). Indexes a codebase into a persistent knowledge graph (tree-sitter, 158 languages) exposed as 15 MCP tools; unusually strong release provenance (SLSA L3, cosign, checksums) but bus factor of 1 on a C binary. Independent testing shows big token savings on structural queries but also that agents default to grep unless actively steered. At BoB repo sizes (~2k files) built-in grep/glob plus the /where skill covers most needs; the unique wins (call-path tracing, diff-impact) are a modest slice. If adopted later: build from source or verify cosign provenance, pin the version, provision per-project via the registry — never curl | bash.

agentsview — wrap (cautious read-only complement)

kenn-io/agentsview (MIT, ~4.4k stars, very active release cadence). Local search + analytics over coding- agent session history; reads ~/.claude/projects/ transcripts into its own SQLite index; web UI with cost breakdowns, heatmaps, full-text search. Genuinely read-only toward the transcripts and loopback by default, but: default-on telemetry, prebuilt binaries with no signing/provenance story, pseudonymous org — and it necessarily reads the entire transcript corpus. Overlaps BoB’s JSONL token accounting (which stays the source of truth) but fills a real gap: full-text/semantic search over months of sessions, which would feed trace-mining far better than raw JSONL greps. Adoption conditions: Docker or source build only, version-pinned, AGENTSVIEW_TELEMETRY_ENABLED=0, loopback-only. Local trial note below.

LMCache — pass, categorically inapplicable

LMCache/LMCache (Apache-2.0, healthy multi-contributor project, production-credible). Manages GPU KV caches inside self-hosted inference stacks (vLLM et al.). BoB’s entire inference path is Anthropic’s hosted API, where the analogous benefit is already delivered by Anthropic prompt caching (which Claude Code uses automatically). No proxy, MCP, or library shape of it can touch a hosted-API workload. Nothing to adopt; the transferable lesson is Anthropic-native cache hygiene — stable prefixes, and tracking cache_read_input_tokens (BoB already records these per session).

flue — pass on adoption; mine as reference

withastro/flue (Apache-2.0, genuine Astro org, pre-1.0). Headless TypeScript harness for autonomous agents with a three-tier sandbox model (in-process bash emulation → local FS → remote containers) behind one interface, a strict session/operation/turn ontology, durable workflows with checkpoint/resume, and structured error objects from a single module. Adopting it would mean rebuilding BoB inside someone else’s harness; the value is architectural reference. Patterns worth filing as BoB issues: (1) a declared sandbox-tier field on dispatched work (formalizes the serial-only pinch-point logic), (2) structured machine-readable errors across hooks/state-machine (today mostly exit codes + text), (3) segment checkpoint/resume semantics modeled on flue’s durable workflows.

headroom: token-savings test on a BoB workload

Rather than inserting the proxy into a live session (see risks above), the compression opportunity was measured directly on representative BoB warp-drive context, 2026-07-17, on this repo:

Sample (content type)	Raw	Reduced	Reduction	How
gh issue list --json ×60 issues (JSON tool output — headroom’s sweet spot)	121.4 KB	11.4 KB	91%	jq field projection
make test full log (repetitive log output)	497.1 KB	7.9 KB	98%	grep to PASS/FAIL/summary lines
state-machine.js (source code)	169.5 KB	—	~15–20%	headroom’s own claimed ceiling for code

Two conclusions. First, the big reductions on BoB’s verbose-tool-output types are real — but achievable with jq filters and log tails inside BoB’s own hooks/skills, with zero new trust surface. Headroom’s differentiated value (ML prose compression, retrieve-on-demand) applies to the content types where its gains are smallest for this workload. Second, the bill shape caps the upside: across all 618 recorded bigbrain sessions, fresh input is 5.1M tokens against 9.04B cache-read tokens (99.5% of the total). Pre-LLM compression shrinks what enters context; the dominant cost is context being re-read, which Anthropic prompt caching already discounts and BoB’s session segmentation (#870) already bounds. A realistic end-to-end saving for a warp-drive session is the ~15–30% band, concentrated in tool output BoB can compress natively.

Action adopted instead of headroom: treat verbose tool output as a first-class cost — prefer jq-projected gh calls and tailed/structured check output in loop instructions (the #586 check-runner already does this for test results).

SkillSpector as a provisioning security gate

NVIDIA/SkillSpector (Apache-2.0, ~1 month public, NVIDIA org, backed by the “Agent Skills in the Wild” dataset — 42k skills analyzed, 26.1% with ≥1 vulnerability). Scans a directory / SKILL.md / repo / zip — exactly the shape of a BoB registry item — for 68 patterns across 17 categories (prompt injection, exfiltration, privilege escalation, supply chain via osv.dev, AST/taint analysis of bundled scripts, MCP tool poisoning). Two-stage: fast static pass, optional LLM semantic pass (~87% claimed precision). CI-ready by design: exit 0/1 on a risk threshold, --format sarif, and a committed baseline file (.skillspector-baseline.yaml) so PRs fail only on new findings.

Proposed BoB gate (filed as a follow-up requirement):

1. CI job alongside make ci: version-pinned install (uv tool install git+…@<commit>, Python 3.12+), scan changed skills/ / registry/ / commands/ / agents/ paths with --no-llm --baseline, upload SARIF, fail on exit 1.
2. Static-only in CI — deterministic, keyless, no content egress.
3. Initial triage sweep of all ~60 items (batch mode, 20 workers) to build the committed baseline. Expect meaningful first-pass noise: BoB’s autonomous-loop skills legitimately pattern-match “excessive agency” rules; baseline discipline is what keeps the gate honest.
4. Provisioning-time hook — cdp / cdprov add shell out skillspector scan <dir> --no-llm for any third-party item, refuse on exit 1. This is where the threat model actually bites.
5. Caveats: no official GitHub Action yet (issue #72) — the gate is a ~20-line hand-rolled step; it augments, never replaces, human review at promotion time; expect rule churn from a month-old project.

Local trial notes

codebase-memory-mcp — works. Trialed 2026-07-17 on this Mac (v0.9.0 darwin-arm64, downloaded via gh release download, SHA-256 verified against the release checksums.txt — no curl | bash). Ran against a scratch repo copy with CBM_CACHE_DIR isolated to the session scratchpad. index_repository --mode fast indexed 92 nodes / 154 edges in 0.2s; search_graph --query walkDir returned the exact function with file/line via BM25. Rough edges: the raw-JSON argument form wants repo_path (not path) and a wrong argument surfaces as a misleading “worker crashed on a file” hint rather than a usage error; the flags form (--repo-path) works cleanly. The binary is 273 MB. Verdict unchanged: functional and fast, but the marginal value over grep//where at BoB repo sizes doesn’t justify a per-session MCP server from a single-maintainer C codebase today.

agentsview — blocked (not trialed). A constrained trial was designed (version-pinned Docker image, telemetry disabled, loopback-only, a sandbox ~/.claude containing only two copied bigbrain transcripts) but the machine’s permission profile denied staging session transcripts for the container — an appropriate outcome given the tool’s posture: unsigned prebuilt binaries from a pseudonymous org whose job is reading the entire transcript corpus, with default-on telemetry. Recommend a human-supervised trial using exactly that constrained recipe: AGENTSVIEW_TELEMETRY_ENABLED=0 docker run --rm -p 127.0.0.1:8991:8991 -v <sandbox-claude-dir>:/root/.claude:ro ghcr.io/kenn-io/agentsview:<pinned>. The fit assessment (wrap, as a read-only viewer; JSONL stays the source of truth) stands pending that trial.

Recommendation

Pilot next (one item): SkillSpector as a static-only CI gate with a committed baseline, plus a cdp/cdprov add scan hook for third-party registry items — filed as a follow-up requirement. It is the only candidate that closes a real gap (no security gate on skills entering the registry) at near-zero trust cost (static-only, keyless, no egress, deterministic).

Deliberate non-adoptions: headroom (re-evaluate in ~2 quarters as a bounded A/B if token cost pressure grows; meanwhile capture the cheap 91–98% wins natively by preferring jq-projected gh calls and tailed check output in loop instructions), codebase-memory-mcp (revisit if BoB repos grow ~10x or call-graph tracing becomes a recurring need), agentsview (pending a human-supervised constrained trial), agent-skills (harvest techniques into existing BoB skills as grooming work — candidate lesson/groom issue), LMCache (inapplicable to a hosted-API stack), flue (reference only; its sandbox-tier, structured-error, and durable-workflow patterns are candidate future BoB requirements).

Per the issue’s note that the roundup is a tip source, not a mandate: five of seven close as “no action now” with explicit revisit conditions.