Test policy (e2e-first)

Test policy (e2e-first)

BoB’s test gates (#1979) decide when checks run: path triggers, tiers, testing_budget_minutes and a proportionate qa_scope. This page decides what a good test is. Warp-drive’s coding and testing phases carry these rules for the resolved policy, and so does the tdd-flow skill.

The policy is the _workflow.test_policy key (see workflow-config.md):

Value	Meaning
e2e-first (default)	All seven rules below.
balanced	Rules 1 and 2 are relaxed: a unit test may be written after the code, for pure logic (a parser, a calculator, a Drupal module’s service class). Rules 3–7 still hold.

Set it per project in the manifest (provisions/<project>.json _workflow) or per checkout in .claude/settings.local.json. bob warp config shows the resolved value and which layer set it (source).

The rules

1. E2E first. Prove behaviour by driving the real entry point (the bob noun or the script) in a scratch directory. Stub only at the edges: gh, claude, the network. Assert what the operator sees: the output, the exit code, the files written.
2. Never unit-test after the code. Write a unit or isolation test only before the code it covers (test-first). A test written after the code is an e2e test.
3. Enumerate failure modes first. Before writing an isolation test, list how the unit can fail. Test those failure modes, not its branches.
4. No tautological or change-detector tests. Do not write a test that checks the code against itself or its own mocks, or one that only mirrors the implementation. Exempt: BoB’s deliberate drift and declarative checks (make check targets, generated-count guards, source-scan checks that enforce the Prime Directive). Those are contracts, not change detectors.
5. Leave a repeatable artifact. Each e2e run leaves something checkable: an output file, a JSON document, a log or a screenshot. The chunk report cites it.
6. A bug’s regression test is an e2e reproduction. The regression gate (#518) is unchanged: the test fails before the fix and passes after. Make it one focused reproduction through the real command. If an existing tests/test-* scenario already exercises the path, extend that scenario instead of adding a file.
7. Keep it fast. Each new test runs in seconds and fits inside testing_budget_minutes.

Examples

Tautological: it asserts the code against its own mock, so it can never fail.

const gh = sinon.stub().returns('[]');
listPlans({ gh });
assert(gh.calledWith(['issue', 'list', '--label', 'flight-plan']));   // restates the call

Change detector: it mirrors the implementation, so it breaks on every refactor and catches no bug.

assert.strictEqual(render.toString().includes('padEnd(12)'), true);

Good e2e reproduction: it runs the real command in a scratch directory with gh stubbed at the edge, and asserts what the operator sees.

// #3199: an overlapping open plan refuses the build
const r = spawnSync('node', [FLIGHTPLAN, 'build', '--issues', '7,9,10', '--json'],
  { env: { ...process.env, PATH: `${stubDir}:${process.env.PATH}` } });
assert(r.status === 3 && JSON.parse(r.stdout).blocked === 'overlap');

Its artifact is the JSON document on stdout plus the stub’s call log. The chunk report names both.

Sources

The rules come from #3093, #3109 and #3110, as decided with the operator on 2026-09-27 and filed as #3200. The halves of those issues that delete tests (#3094, and the cleanup command in #3109) are out of scope.