izri/hallucination

Hallucination answers one question: is what this change says about itself true?

A hallucination is something a change says about itself that the evidence doesn't back up.

AI-written code reads well and looks plausible. A pull request makes claims in three places — its code, its tests, and the text around them — and each can be checked against something Izri can actually get: the repository, the package registry, and the results of running the tests.

Kind	What the change claims	Evidence
Reference	"This thing exists." An import, an exported symbol, a package, an env var.	The repository tree, manifests, the package registry.
Verification	"These tests prove the change works."	The tests run with the change and with it reverted.
Description	"I did X." The PR body and commit messages.	The diff.

A claim doesn't have to be written down. Opening a pull request claims "this works", so a change with no description can still hallucinate.

What this signal is not:

• Not test results or coverage. Whether the tests pass, whether they ran the changed lines, and whether they assert anything is izri/tests.
• Not scope. Scope asks whether the change matches what was asked. "Says fix, rewrites auth" is scope; "imports a package that doesn't exist" is hallucination.
• Not an LLM's opinion. See below.

Tools decide, models point

One model grading another shares its blind spots, so Izri doesn't do that. A model may translate prose into a checkable claim or point at what to check; a deterministic tool — a resolver, a registry lookup, a test run — decides.

• A finding backed only by a model's judgment is capped at warn, whatever severity the model gave it.
• Every finding carries its evidence: which tool, what it saw, and where.
• No category blocks a merge until it has measured precision on real pull requests.

The revert check

Coverage says the tests ran the changed lines. It cannot say whether they checked anything. The revert check can: would the tests still pass if the change were reverted? A test that passes at the PR head and fails with the change reverted depends on the change. One that passes both ways verifies nothing about it.

The check runs in your CI, not on Izri's machines, with izri verify (or the GitHub Action's mode: verify). It picks the tests that relate to the diff — tests the PR added or changed, and tests that import a changed source file — runs them once at head and once with the changed source files restored to the merge base, each in a throwaway git worktree, and uploads both JUnit results. Izri executes nothing.

It is judged against izri/tests evidence for the same commit. A revert-check upload is never izri/tests evidence itself: that comes only from the Izri runner or an izri upload, whichever the project takes its tests evidence from. Until that evidence exists for the head, the upload is stored and izri/hallucination says Revert check not run: no izri/tests evidence for this commit. When the evidence lands, the stored upload is judged — the two can arrive in either order, nothing is uploaded twice. A red izri/tests run does not stop the check; its summary then notes that the head run was not green. A project that has used izri verify before is told when a commit has no revert-check upload; one that never has sees nothing about it.

Head	Reverted	Meaning
pass	fail	Depends on the change. Supporting evidence.
pass	pass	Does not detect the change.
pass	error loading the module	Inconclusive — it needs the file to exist, not to behave.
fail or error	anything	Excluded from the revert check. A failing test is reported by izri/tests, where it fails the verdict.

Evidence a PR could have shaped is inconclusive. The revert-check upload is vetted by the same rule as izri/tests evidence: Izri asks GitHub — never the upload — which files the PR changed, and the GitHub Actions id-token sent with the upload names the workflow that produced it. If the PR edited that workflow (or, for a reusable workflow, the file whose job ran), a composite action, a test-runner or build config, a test setup file, a Makefile, the test scripts in package.json, or .izri/, the results are reported as verification_evidence_untrusted and neither support the change nor count against it. The same applies to an upload without an id-token (izri verify outside GitHub Actions, or a job without permissions: id-token: write), and when no GitHub App installation can confirm the changed files.

Finding	Severity	Meaning
change_unverified	warn	Every selected test still passed with the change reverted: nothing depends on this change.
verification_evidence_untrusted	warn	The PR edited what produced the evidence, or nothing proves which workflow did, so the revert check is inconclusive.
verification_skipped_budget	info	Too many tests were selected for the revert check; nothing ran.

The revert check also names individual tests this PR wrote that pass either way (test_passes_without_change). That is a statement about a test file, so it is reported under test quality on izri/tests, advisory like the rest.

A flaky failure in the reverted run looks like support, so flakes suppress findings rather than invent them. A PR made only of new files gets little from this check — deleting a new file breaks the import, which is inconclusive — and relies on coverage and the static checks instead.

Reference checks

Coverage answers whether the tests ran the change. The reference checks answer an earlier question: does the code the change refers to exist at all? An import of a file that isn't there, a named import the module doesn't export, a package nobody declared or that the npm registry has never seen, an env var nothing declares. This is the most literal way an agent hallucinates in code, and it needs no tests to catch.

They run on every push, server-side, against the repository tree at the PR head. Nothing is installed or executed. Scope today: TypeScript/JavaScript and npm.

Category	Severity	Meaning
import_unresolved	warn	A relative or tsconfig-aliased import on an added line matches no file in the repository.
export_missing	warn	A named import from a first-party module that the module does not export.
dependency_undeclared	warn	A bare import of a package that no package.json above the file declares.
package_not_found	warn	A dependency this PR added does not exist on registry.npmjs.org.
package_suspicious	warn	A dependency this PR added is under 30 days old with fewer than 1,000 weekly downloads — the shape of a typosquat or an invented name.
env_undeclared	info	An env var read on an added line that appears in no env template, env config or workflow.
reference_unparsed_import	info	An import whose specifier is not a string literal was not checked.
reference_file_unparseable	info	The file has syntax errors, so the parser may have missed imports.

Every finding carries tool evidence (the resolver, the manifest, the registry response) and a file and line. None of them blocks yet; they ship at warn until their precision is measured on real PRs.

What your CI already catches. A repo whose CI type-checks and installs dependencies will already fail on an unresolved import, a missing export or an unknown package. These checks still help JavaScript repos, repos without a type-check step, and anyone who wants the answer before CI finishes. package_suspicious is the one CI cannot give you: a freshly registered or typosquatted package installs cleanly.

What it refuses to guess. A repository tree GitHub truncates, a module that could not be fetched, a registry that did not answer, a package routed to a private registry in .npmrc, a workspace package, or a module that re-exports with export * — each of these produces no finding rather than a possibly wrong one.

Scoring

The score runs 0–100 and carries weight 0.40 in the composite by default. It starts at 100 and loses 25 per error, 10 per warn and 2 per info finding. A check that ran and found nothing is a real 100; when no hallucination check applies to a delta, the row is skipped and its weight is redistributed.

If one stage of the check crashes, the others still report and the crash is listed as analyzer_stage_error (info). If every stage crashes, the signal fails — a crashed analyzer is not a clean one.

Drilling in

check_hallucinations   # findings for a delta, grouped by the signal that judges them
check_test_efficacy    # same shape, addressed by test_run_id

Both are MCP tools.

Related

• izri/tests — pass/fail, diff coverage, and the quality of the tests a PR wrote.
• Signals overview — the aggregation model.