Tracer Classifier Evaluation

Eval: the tracer classifier

Eval for #2738, part of vertical-slice-first decomposition (#2733), on the signal chosen by decision #2742. The precedent is the dedupe pass (#2681).

Status: the threshold is provisional. The classifier ships with a threshold of 0.85 on the probability. That is the dedupe pass’s bar, carried over and not yet tuned. The live Jev run against the labelled set below has not happened, because the machine that built the classifier has no Jev key. Until it runs, treat a tracer label the classifier applied as a suggestion the operator confirms.

What is measured

scripts/warp-drive/tracer-classify.js asks one noul through the decide() seam (caller tracer-classify in scripts/lib/decide-callers.json): does the work this requirement describes change more than one layer of a running system, or build a new feature from nothing? Only the title and the first 1200 characters of the body leave the machine.

Probability p	Verdict	With --apply
Confidence abs(2p − 1) below the seam’s 0.5 floor	ask	One comment asks the operator.
p ≥ threshold	tracer	Adds the label, plus one comment with p, the threshold, the model and the layer cues.
p ≤ 1 − threshold	single-layer	Nothing.
In between	below-threshold	Nothing.

Without --apply it only reports. It never removes a label. With no key, or with Jev unreachable, it prints tracer: skipped and exits 0.

The labelled set

tests/fixtures/tracer/cases.json holds 20 synthetic, operator-neutral cases: 10 cross-layer or greenfield and 10 not. The negatives are a rename, a typo fix, a dependency bump, test-only work, a CSS tweak, an index-only migration, a refactor, a docs page, output sorting and a constant change. The positives each name at least two layers, or a new build from nothing. Four of them are deliberately hard for a keyword rule: a new CLI command, a rate limit that spans the Worker and the UI, a state-file field surfaced in status output, and an onboarding page.

Results

Backend	Model	Threshold	Precision	Recall	Ask	Date
deterministic (layerRule, keyword cues)	—	0.85	100% (7/7)	70% (7/10)	0	2026-09-29
jev	jev-1.13.0	to tune	pending	pending	pending	—

The deterministic rule is the offline baseline and the test backend. It calls a case cross-layer when it finds cues for two or more layers, or a greenfield cue. It misses the CLI-command, rate-limit and state-to-status cases, because their layers are named in words it does not know. That is the gap the model is there to close. Do not tune the keyword lists to this set, because that would only measure the set.

Running the live tune

On a machine with TYPESAFE_API_KEY (check with node scripts/lib/jev.js status):

node scripts/warp-drive/tracer-classify.js eval \
  --cases tests/fixtures/tracer/cases.json \
  --record tests/fixtures/tracer/jev-1.13.0.json --json > /tmp/tracer-eval.json
jq '.summary | {model, precision, recall, ask, sweep}' /tmp/tracer-eval.json

Choose the lowest threshold in the sweep whose precision is at least 90% (the eval’s bar, PRECISION_BAR). Set DEFAULT_THRESHOLD to it, fill in the jev row above with the model version from the summary, and commit the recorded fixture so later runs can replay it offline with --fixtures. The threshold does not carry across Jev versions (scripts/lib/jev.js), so re-run this whenever JEV_MODEL changes.