vuln-hunt Evaluation

vuln-hunt Evaluation (#2668)

Evaluated 2026-10-01 (by reading) and 2026-10-11 (vendoring and the sandbox), against cloudflare/security-audit-skill at commit c1c8a8c1471069fb0e188eeaff69b8e8db6564a8 (MIT). The operator chose vendor and adapt (option A on #3545). The skill is vendored as registry/skills/vuln-hunt, with the upstream files unchanged and BoB’s layer in its BOB.md.

What it is

A six-phase audit run by a parent agent that delegates to isolated agents: reconnaissance, coverage-led hunting, validation by a fresh agent that tries to disprove each candidate, structured findings checked against a JSON schema, independent verification of every final record, and a target-neutral report. Two zero-dependency Node validators (validate-coverage-ledger.cjs, validate-findings.cjs) are the gates between phases. A finding is confirmed, needs_validation (an exact unresolved fact, no severity) or rejected.

BoB had no audit of this kind. The security-auditor agent is one reviewer making one pass, and /security-review covers a diff. vuln-hunt sits beside them; it does not replace them (see BOB.md’s “When to reach for it”).

Compatibility with BoB’s agent model

Requirement	In Claude Code / BoB	Fit
A parent that owns shared state and delegates	the main session, with the Agent tool	fits
research agents (read-only exploration, fact checks)	subagent_type: Explore	fits
general agents (investigation, bounded local execution)	subagent_type: general-purpose; execution only through the sandbox helper	fits
Several agents in parallel (reconnaissance, Phase 5 verifiers)	several Agent calls in one message	fits
Write isolation: each agent writes only its own scratch/	by prompt, not enforced: subagents share the filesystem and tools. The parent is the only writer of the ledger and findings, and a malformed or out-of-place result is discarded (the skill’s own rule)	gap, mitigated
Agents return one JSON object, no prose	agent results are text; the parent parses and discards anything else, as the skill says	fits
Model	none named upstream. The hunters and verifiers do the reasoning the skill exists for, so BoB runs them on the reasoning role (bob route roles), never the cheap one	fits
Budget	counted in agent invocations, reserved up front for critics and verification (run-metadata.json budget). A warp run’s --budget is dollars; the two are separate ceilings, and the stricter stops first	fits, two ceilings
Output outside the target	default ~/security-audit-skill/<repo>/run-<N>, kept as is	fits
No dependency installs, no network to live services	the sandbox has no network and the helper never pulls an image	fits

The cost is the real constraint. A standard run on a repo of BoB’s size is dozens of agent invocations, so it belongs to the security-review ritual (#2422) or an explicit request, never to every PR.

The sandbox requirement

The skill runs target-controlled code only inside an OS-enforced sandbox with four controls: no external network; an empty environment built from an allowlist; a read-only target and toolchain, with writes only in scratch; and CPU, memory, process, file-size and wall-clock limits. If any control cannot be enforced, the code is not run and the lead stays needs_validation.

• Unprivileged user namespaces are blocked on farm-01 by AppArmor (unshare -Urn true fails with “Operation not permitted”), so bubblewrap and similar tools are out.
• Docker meets all four. scripts/security/vuln-hunt-sandbox.sh starts every container with --network none, a read-only root, the target mounted read-only at /target, only /scratch and a 64 MiB /tmp writable, no host environment (HOME, TMPDIR and caches point into scratch; --env adds an allowlisted value), --cap-drop ALL, no-new-privileges, the caller’s uid, memory, process, CPU, file-size and open-file limits, and a wall clock that kills the container (exit 124). check proves each control in a throwaway container and reads the limits back from Docker’s own record. tests/test-vuln-hunt-sandbox.sh drives both against the real daemon.

Gaps, stated rather than hidden:

• The image. The helper never pulls, so the toolchain image the target needs (Node, PHP, …) must already be on the machine. On farm-01 today only the DDEV images are present: enough to prove the controls, not to build a Node project. Choosing and building a toolchain image is a pull, so it is the operator’s call. Until then, a lead that needs Node stays needs_validation.
• Docker-group access is root-equivalent on the host. The sandbox protects the host from the target code. It does not make the agent’s own Docker access safe.
• Machines without Docker. Wherever Docker is not installed or its daemon is not reachable (a laptop without Docker Desktop, for one), the helper answers “unavailable” (exit 3) and the run stays source-only, which the skill handles: leads stay needs_validation.

Patterns worth reusing in BoB skills

These are what the skill is worth beyond its own audit. Each names where it would land.

1. A coverage ledger with a validator as a gate. Every unit of work is a ledger row with an owner, a state and its evidence. After every update a deterministic validator runs, and an invalid ledger cannot drive the next assignment. In BoB: a flight plan is already a ledger without the gate. The swarm gate and /groom passes could carry the same “validator between phases”.
2. The checker is never the finder. A fresh agent that did not hunt tries to disprove each candidate. A material change to a record gets another agent that touched none of the earlier steps. In BoB: the graph-orchestration skill’s “diamond + checker”, and /review-team’s role split. vuln-hunt adds the rule that a promotion to a stronger verdict always needs a new verifier.
3. Three verdicts, and needs_validation has no severity. Uncertainty gets its own state with the exact fact that would settle it, instead of a low-severity guess. In BoB: security-auditor’s “Low / needs-verification” could become this verdict, and so could /code-review’s uncertain findings.
4. Budget counted in invocations and reserved up front. The critics and the verification that make the result trustworthy are funded before the first hunter starts. A run that cannot fund them does not start, and a run that runs out stops as incomplete with a named reason. In BoB: fan-out in bob swarm and Workflow scripts spends first and checks later.
5. Additive runs keyed by fingerprint. A later run reads the earlier ledger and findings, targets the gaps and re-checks changed source, and never counts stale work as covered. In BoB: the issue fingerprints that harness-defect-report.js and the quarantine bugs already use; vuln-hunt’s issues carry theirs the same way.
6. A malformed result is discarded, not repaired. The parent re-runs the agent instead of fixing its output, so it never writes evidence of its own into a record. In BoB: every place a parent parses a subagent’s JSON.

Status

• Vendored, in the security kit, documented (#2668, chunk 1).
• Sandbox helper and its test (#2668, chunk 2).
• Not yet run. Executing the vendored code (its validator tests, then an audit of a sample repo and of bigbrain) was refused by this session’s auto-mode safety check, which treats it as third-party code. It waits on the operator; see #2668.