Jev Model Router Evaluation

Eval: jev-model-router

Eval for #2667, part of the LLM cost-tiering work (#1974). It builds on the model-assignment policy (#2680), subagent-aware token accounting (#2685) and the Jev client and egress policy (#2672).

Verdict: no-go for now. BoB does not install the mod on any machine. The reasons are below, together with the conditions for a re-evaluation.

What was reviewed

The mod’s source is davila7/claude-code-templates, at cli-tool/components/mods/productivity/jev-model-router/ (upstream commit 666c061, reviewed 2026-09-19). It has three parts:

• hooks/jev-model-router.ts registers the function hooks.
• hooks/policy.ts holds the routing policy.
• .claude-plugin/plugin.json declares the options.

The mod needs Claude Code 2.1.259 or later and CLAUDE_CODE_ENABLE_FUNCTION_HOOKS=1. The function-hook $ API is early access.

The mod hooks three events:

Event	What it does
prompt.submit	Classifies the user’s prompt: tier, effort and risk.
turn.step	Applies the result to the main loop: its effort, and its model if routeMainModel is on.
agent.spawn	Classifies the subagent’s prompt, description and type, and sets the subagent’s model.

Every failure path fails open: the request goes ahead exactly as the engine built it.

The live sandbox run (claude plugin validate, then a headless session with the mod loaded) was not done in the warp-drive run: executing downloaded third-party code needs a person’s approval. The findings below come from reading the code. The sandbox run is tracked as a todo on #2667.

Findings

1. The built-in classifier path cannot save anything

With no API key, the mod calls the engine’s $.model.classify(prompt, ['fast', 'balanced', 'deep']). The result has a tier and nothing else: confidence, effort and risky are all null. policy.ts then does two things:

• It never moves effort. route() skips effort when the effort score is null, so routeMainEffort does nothing on this path.
• It only upgrades a model. allowed() returns !isDowngrade when confidence is null. That is the mod’s own safety rule: spending less on an unmeasured guess is the bad trade.

So on the path that keeps prompts on the machine, the only change the mod can make is to move a subagent to a more expensive model. A trial of this path can measure a cost increase, never a saving.

2. It overrides the declared agent tiers

#2680 sets each agent’s model: to follow its work class, and a warp-drive delegation spawns haiku or sonnet on purpose. agent.spawn measures the change from the requested model (e.model ?? e.parentModel), so the router second-guesses a declared choice. On the built-in path a haiku mechanical subagent classified deep goes to opus, which undoes the lever #2680 measured. The model-assignment policy is declared and reviewed in version control. A per-prompt classifier that overrides it is a second source of truth for the same decision.

3. It mis-ranks models it does not know, and that can move a subagent down unmeasured

rankOf() finds a model’s tier by substring: the configured tier names, then haiku / sonnet / opus. Anything else ranks null, and claude-fable-5-1 is one of those. With the current rank null, allowed() does not treat any move as a downgrade. A subagent that would inherit a Fable parent can therefore go to haiku or sonnet on an unmeasured built-in label. The confidence-null guard exists to stop exactly that. The fix is upstream: an unknown current model should rank as the top tier, or the tier map should be configurable. Until then, the “only routes up when unmeasured” guarantee does not hold on a Fable session.

4. The hosted path conflicts with BoB’s egress policy

With a TypeSafe or Gateway key, prompt.submit sends { prompt: e.text }, the whole user prompt, and agent.spawn sends the subagent prompt and description. The llm-tiering egress policy (#2672) requires filtering in code to the fields the questions read, with no secrets ever. A raw prompt is unfiltered: a pasted token, a log line or a .env fragment would leave the machine. The mod has no filter hook.

Privacy decision: BoB does not enable a hosted backend for this mod. Revisit only if upstream adds a prompt filter that BoB can supply, such as a redaction callback or a length and field allow-list.

5. It defaults to an unpinned model

typesafeModel defaults to jev-latest. BoB pins jev-1.13.0 because an alias moves when TypeSafe ships and thresholds do not carry over. A config can override this, so it does not block adoption, but any adoption must set it.

6. routeMainModel stays off

The mod reaches the same conclusion as BoB’s own measurement in Warp-Drive Model Routing: 67–81 % of per-chunk cost is cache reads of the accumulated context, and switching the main-loop model mid-session re-caches all of it. The mod ships with routeMainModel off, and BoB keeps it off.

What BoB already does instead

Decision	Where BoB makes it
A subagent’s model	The agent’s model: frontmatter, set from the model-assignment policy (#2680)
Delegating a mechanical chunk	_workflow.delegate_mechanical_chunks with the three delegation tests; the main loop is the fail-open backstop
A typed classify, route or dedupe decision in a pipeline	scripts/lib/jev.js: pinned version, confidence tiers, filtered state, fail-open (#2672)
Main-loop reasoning effort	The warp-drive engine’s per-phase REASONING BUDGET line

The per-task classifier that the mod adds on top of these would compete with the declared policy rather than fill a gap. The mod’s safety model is worth reusing if BoB builds its own router: asymmetric upgrade and downgrade bars, upgrades only when unmeasured, a risk override and fail-open on every path. The llm-tiering skill already follows the same shape.

A/B measurement

The planned A/B was ten chunks with routeSubagentModel and routeMainEffort on, against the #2680 baseline. It is not run. On the built-in path, finding 1 settles the outcome in advance. On the hosted path, finding 4 rules the trial out before any measurement. If a re-evaluation condition below is met, the A/B runs then, over subagent-aware token records (#2685).

Re-evaluate when

Any one of these reopens the evaluation:

• Upstream adds a prompt filter BoB can supply, so the hosted path can meet the egress policy.
• The mod can respect a requested model: a switch that routes a subagent only when the caller named no model. The declared agent tiers would then stand.
• rankOf() treats an unknown model as the top tier, or the tier map becomes configurable, so Fable sessions are safe.

If BoB adopts the mod after a re-evaluation, it is declared: the mod is vendored under registry/ at a pinned upstream SHA, and the environment flag and pluginConfigs are set through BoB-managed settings. It is never hand-installed into ~/.claude/settings.json.