izri/tests

Tests answers one question about a pull request: are the tests for this change any good? That breaks into three:

1. Do they pass?
2. Do they run the lines this change touched?
3. Do the tests this PR wrote check anything?

Izri judges the delta a PR introduces. The rest of your project is context for that judgement — it is never what's being graded. So coverage here always means diff coverage: of the lines this PR added or modified, how many did a test execute. Project-wide coverage is your coverage tool's job.

1. Pass / fail

A failed run fails izri/quality, whatever the other signals say. One red test must never read as "safe to merge".

Finding	Meaning
test_failure	The run finished with failing tests. Hard rule. The failed tests are listed on the report.
runner_error	The run broke before producing results. Hard rule — no evidence never resolves to "safe".
zero_tests	A passing run executed no tests. Hard rule, opt-in via block.zero_tests.
evidence_missing	The project uploads from CI, and nothing arrived for this commit within the grace window. Hard rule.
test_evidence_untrusted	Green evidence from a PR that edited what produced it. warn; hard, opt-in via block.untrusted_evidence.

Run statuses

Status	Meaning
PENDING	Queued, not yet picked up.
RUNNING	Executing.
PASSED	Terminal. Everything passed.
FAILED	Terminal. Tests failed.
ERROR	Terminal. The run itself broke.
CANCELLED	Terminal. Stopped before completion.

PASSED is the only terminal status that isn't a failure. The CLI exits non-zero on every other one.

2. Diff coverage

A green suite says nothing about whether it touched your change. Here is the case this exists for.

An agent is asked to fix a rounding bug in calculateTax. It changes three lines and adds tax.test.ts with four new tests. CI is green.

// src/billing/tax.ts — the change
export function calculateTax(cents: number, rate: number): number {
-  return Math.round(cents * rate)
+  return Math.floor(cents * rate + 0.5)
}

// tests/tax.test.ts — the new tests
import { formatTax } from '../src/billing/format'

it('formats a tax amount', () => {
  expect(formatTax(1050)).toBe('$10.50')   // never calls calculateTax
})

Four passing tests, none of which call calculateTax. Project coverage barely moves. Izri reports change_uncovered and the verdict fails: a suite that grows without touching the diff manufactures confidence.

Each changed file lands in one bucket — covered, partial, uncovered, or notInstrumented (the coverage tool never saw the file: a deliberate exclusion, or a real gap; Izri says so at info rather than guessing). Uncovered lines are annotated inline on the pull request.

Finding	Severity	Meaning
change_uncovered	error	A changed file where no test executed any changed line. Hard rule.
change_partially_covered	warn	Some changed lines were executed, some weren't.
low_diff_coverage	warn	Under 60% of the added lines were executed. Named low_overall_coverage before September 2026.
file_not_instrumented	info	The coverage tool never saw the file.
diff_coverage_below	error	Diff coverage is under your project's minimum. Hard rule, opt-in.

Coverage minimum

Partial coverage doesn't block by default — real code often has a branch the tests don't reach. To require more, set a minimum in the project's merge policy:

policy:
  block:
    diff_coverage_below: 80

A delta whose diff coverage is under 80% then fails with diff_coverage_below. It is off by default, it never fires when coverage wasn't measured, and it stays quiet when change_uncovered already fired for the same delta — one fact, one blocking finding.

3. Test quality

Findings about the tests this PR wrote. They are advisory: shown on the report, never scored, never blocking. Heuristics about someone's assertions are opinionated, and a false "your tests are weak" that blocks a merge is how a gate gets switched off. A category is promoted only after it has measured precision on real pull requests.

Finding	Severity	Meaning
change_lacks_test	warn	A changed source file has no changed test importing it. Static; a nudge, not a verdict.
weak_assertion	warn	An LLM judged the assertions too weak to catch a regression. Listed separately as LLM-derived.
assertion_free_test	warn	A test block this PR touched contains no assertion.
tautological_assertion	warn	An assertion that cannot fail, such as expect(true).toBe(true).
test_mocks_subject	warn	A test mocks a module this PR changed and no test file in the PR imports that module unmocked, so the change may never run under test. Mocking a changed collaborator is fine when its own test is in the PR.
test_passes_without_change	warn	A test this PR wrote passes with the change reverted. Found by the revert check.

Scoring

The tests score is the lower of pass rate and diff coverage, 0–100, and carries weight 0.36 in the composite by default. Green tests that ran half of the change score 50, not 100. With no coverage measured it is the pass rate alone.

Where the evidence comes from

Most teams already run their tests in CI, so Izri doesn't run them again. Each project has one evidence source, set in the dashboard or with tests.evidence in .izri/config.yml:

Source	What happens	When to use it
ci_upload	Your CI runs the tests and uploads the JUnit results, plus a coverage report, with izri upload. Izri judges that run.	Any framework, any CI. The default once a project has made a trusted upload.
runner	Izri's runner executes the tests relevant to the diff and collects coverage itself. Playwright only.	Projects with no CI of their own.
none	No tests verdict. The row shows as skipped and its weight goes to the other signals.	New projects until they pick one; the first trusted upload switches to ci_upload.

Uploading from CI

jobs:
  tests:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write # lets Izri trust the upload
    steps:
      - uses: actions/checkout@v4
      - run: npx vitest run --reporter=junit --outputFile=junit.xml --coverage --coverage.reporter=lcov
        continue-on-error: true
      - uses: legendify-dev/izri/apps/github-action@v1
        with:
          api-token: ${{ secrets.IZRI_API_TOKEN }}
          mode: upload
          junit-file: junit.xml
          coverage-file: coverage/lcov.info

or izri upload --junit junit.xml --coverage coverage/lcov.info from any CI. LCOV and Cobertura coverage are accepted. Always upload, even when nothing ran: izri upload --skipped "docs-only change" records an explicit empty run. Silence is treated as a broken step, not as a clean one.

When the upload doesn't arrive

A project on ci_upload whose CI never uploads for a commit holds izri/quality at unknown for a grace window (six hours by default), then fails with evidence_missing: no evidence never resolves to "safe", and a failing check with a cause beats a pending one without.

Evidence the PR could have shaped

A pull request that edits the workflow that produced the evidence, a composite action, a test-runner or build config (vitest.config.*, jest.config.*, playwright.config.*, tsconfig*.json, babel.config.*, pytest.ini, pyproject.toml, …), a test setup file, a Makefile, the test scripts in package.json, or .izri/ could make its own tests look green. The list is a heuristic and cannot be complete; it catches the common ways, not every way. Izri checks with GitHub — never with the upload — what the PR changed.

Which workflow produced the evidence comes from the GitHub Actions id-token sent with the upload in the X-Izri-OIDC header. The job needs permissions: id-token: write; izri upload and the Action request and send the token themselves, and the REST reference shows how to send it when calling the route directly. Its signed claims name the repository, the commit, the run and the workflow file, and for a reusable workflow the file whose job ran. Only those workflow files taint when edited; editing any other workflow changes nothing.

Green evidence is trusted only when all of this holds:

• the upload carries a valid id-token for the project's repository — without one (a local izri upload, another CI system) it is stored and judged but never trusted, because anyone holding a project token could have sent it for any commit;
• GitHub confirms the run is of that workflow on this commit;
• a GitHub App installation on the repository could list what the PR changed, and the PR changes fewer than 300 files (past that GitHub cannot list them all, so they cannot be vetted);
• the PR edits nothing on the list above.

Red stays red whoever produced it. Untrusted green is still scored but carries a test_evidence_untrusted warning giving the reason; the opt-in block.untrusted_evidence policy rule makes it block instead.

On the delta report, the izri/tests card and its test-run dialog show who produced uploaded evidence: Produced by <workflow file> (run #n), linking to the GitHub Actions run, with the job's workflow added when a reusable workflow ran it, and the actor and event in the dialog. Green evidence that is not trusted carries the reason: in the dialog, and on the card unless its test_evidence_untrusted finding already says it. A red run, or an upload the verdict did not use, names its producer and nothing about trust.

The runner

With runner, runs execute on Izri's runner — the queue consumer that drains jobs from Redis — so a self-hosted instance without a runner leaves runs at PENDING indefinitely. See self-hosting. izri run --watch triggers one and polls until it is terminal; the GitHub Action does the same with mode: tests. A project on another source still gets a runner job when izri/visual needs screenshots; that job is labelled "visual capture" and is not the tests verdict.

What to do when it fails

• Red tests: fix them; the report names each one.
• change_uncovered or a coverage minimum: write a test that calls the changed code. Not more tests — the right test. The inline annotations show exactly which lines went unexercised.

Related

• izri/hallucination — whether what the change says about itself is true.
• Verdicts and exit codes.
• Troubleshooting — stuck at PENDING.