izri/hallucination

Hallucination answers the question a green build cannot: did the passing tests actually exercise the code that changed?

This is the least intuitive signal and the most valuable, so start with the case it exists for.

A worked example

An agent is asked to fix a rounding bug in calculateTax. It changes three lines. It also adds tax.test.ts with four new tests. CI runs. Everything is green.

Look closer:

// src/billing/tax.ts  — the change
export function calculateTax(cents: number, rate: number): number {
-  return Math.round(cents * rate)
+  return Math.floor(cents * rate + 0.5)
}

// tests/tax.test.ts  — the new tests
import { formatTax } from '../src/billing/format'

it('formats a tax amount', () => {
  expect(formatTax(1050)).toBe('$10.50')   // never calls calculateTax
})

Four passing tests, none of which call calculateTax. Coverage across the project barely moves, because the file was already covered by older tests that don't reach the modified branch. izri/tests reports PASSED. Nothing is wrong, as far as any conventional signal can tell.

izri/hallucination reports change_uncovered, and the umbrella fails.

That is the entire point. A test suite that grows without touching the diff is worse than no new tests, because it manufactures confidence.

What it measures

Diff coverage, not project coverage. For every added or modified line, did the test run execute it?

Each changed file lands in one of four buckets:

Bucket	Meaning
covered	Every changed line was executed.
partial	Some changed lines were executed, some weren't.
uncovered	No changed line was executed.
notInstrumented	The coverage tool never saw this file at all.

notInstrumented is ambiguous on purpose. It might be a deliberate exclusion in your coverage config, or a real gap where the test stack never imports the file. Izri reports it at info rather than guessing.

Findings

Category	Severity	Meaning
change_uncovered	error	No test exercised the diff. Hard rule — fails the umbrella outright.
change_partially_covered	warn	Some changed lines were exercised, some weren't.
low_overall_coverage	warn	Project coverage is low enough to weaken the signal.
file_not_instrumented	info	The coverage tool never saw the file.

Only change_uncovered is absolute. Partial coverage is a soft deduction — real code frequently has a branch the tests don't reach, and blocking on that would make the signal unusable.

Scoring

coverage_score runs 0–100 and carries the heaviest weight in the composite, 0.40. A partially covered diff should pull the verdict down even when scope and tests both look clean, because a partially tested change is exactly where regressions hide.

Drilling in

check_hallucinations   # per-file breakdown for a delta
check_test_efficacy    # same shape, addressed by test_run_id

Use check_test_efficacy right after triggering a run when you want the verdict for that specific attempt rather than the latest one on the delta. Both are MCP tools.

What to do when it fails

Write a test that calls the changed code. Not more tests — the right test. The per-file detail lists exactly which lines went unexercised, so the target is unambiguous.

Related

• izri/tests — whether the suite passed at all.
• Signals overview — the aggregation model.