izri
izri
03 / 04izri/hallucination

Are the tests real — or did the agent invent them?

AI-written tests can assert against APIs that don't exist, mock the very thing they claim to verify, or pass vacuously. Izri traces every assertion back to a real symbol in your codebase — a fabricated one is a hard fail, no matter the other scores.

  • Every asserted symbol resolved against the real code graph
  • Detects self-mocking & tautological assertions
  • Fabrication is a hard fail — it vetoes the umbrella
assertion trace · checkout.test.tshallucination · hard fail
✓cart.total()resolves → lib/cart.ts:41
✓applyCoupon()resolves → lib/promo.ts:18
✗api.users.bulkVerify()no such symbol
↳ assertion targets a method that exists nowhere in the codebase

What it asks

Intersects the per-line v8 coverage captured by the test run with the PR diff. Lines that were added but never executed are "uncovered" — by definition the tests cannot have detected a regression on them. A "change_uncovered" hard rule fires when zero added lines were exercised at all, because the test pass is then meaningless for this delta.

Hard rules

Auto-fail
  • •change_uncovered — zero added lines were executed by any passing test.

Soft signal

hybrid

Coverage score 0–100 = percentage of added lines that were exercised by at least one passing test. Composite weight in izri/quality: 0.40 — the strategic differentiator, highest single weight. A partially-covered diff drags the umbrella down even when scope and tests look clean.

When it fires

Computed by the runner alongside the test suite when `runtime.coverage` is wired to a coverage file path (lcov, monocart, v8). Without coverage data the hallucination check skips and the umbrella aggregator redistributes weight to the remaining signals — no penalty for projects that haven't wired coverage yet.

Implementation reference

The docs page has the full type signatures, config schema, and the on-disk shape of every analyzer output. Or grab the raw markdown via /docs/features/test-generation.md.

Read the docs
Get startedizri/qualitySee all four signals