Test framework support

Resolution of #267. The framework gate is a code-level constant by design — this doc explains why and what to do when adding a new framework.

Current support matrix

Only Playwright is supported today. The other entries are placeholders — see Adding a framework below for what shipping each would involve.

How the gate works

A project participates in the izri/tests signal only if it declares a supported framework on its projects.test_frameworks column. That column is populated by the repo-analyzer (analyzeRepository) and can be backfilled on existing projects via the bulk-backfill admin procedure (see #266 / analysis.backfillTestFrameworks).

The gate itself is one constant in packages/trpc/src/services/deltaTestTrigger.ts:

const SUPPORTED_FRAMEWORKS = new Set(['playwright'])

triggerTestRunForDelta checks each delta's project against this set and returns { triggered: false, reason: 'no_supported_framework' } for anything that doesn't match. Frameworks are compared case-insensitively.

Why code-level rather than per-project setting

The expansion path discussed in PR #211 considered moving the gate to a per-project setting backed by a framework registry. We landed on code-level for the following reasons:

1. Adding a framework is always a code change anyway. The runner (@izri/test-executor) needs to know how to discover specs, install dependencies, and invoke the framework binary. That's per-framework code, not configuration.
2. No customer is currently asking for it. 100% of shipped customers are on Playwright. Building a runtime registry now would be designing against zero data points.
3. The constant is auditable. Every PR that wants to expand framework support touches one obvious file. Reviewers can spot it. A registry that "just works" when admins flip a feature flag is much harder to reason about.
4. Migration is cheap. When we have two or three frameworks shipped, we can convert the constant to a registry without breaking anything — the wire format on the project row already allows multiple values.

Revisit this decision when:

• A paying customer needs a non-Playwright framework on a deadline that the code-level expansion can't meet.
• The runner gains the ability to install + execute a framework dynamically (today every framework requires bespoke runner code).

Adding a framework

Expansion checklist when a new framework lands. Treat each step as gated — don't merge a half-built path.

1. Runner support (@izri/test-executor)

• Add an executor implementation in packages/test-executor/src/. It must match the existing DockerPlaywrightExecutor shape: dispose, run, parse results into the canonical TestRunResult.
• Add framework detection in packages/repo-analyzer/ so analyzeRepository populates projects.test_frameworks correctly for the new framework.
• Choose the docker base image and the install/exec commands. Document any framework-specific environment assumptions (Node version, etc.).

2. Spec selector (specSelector)

• Update packages/trpc/src/services/specSelector.ts (or its sibling test selector logic) to understand the framework's spec-discovery rules. Playwright treats positional args as file globs; Vitest does similar but with different defaults; Jest needs --testPathPattern; etc.
• Update TestSelectionPlan to carry framework-aware arguments if the new framework needs flags Playwright doesn't (e.g. --config paths).

3. The framework gate

• Add the framework name (lowercase) to SUPPORTED_FRAMEWORKS in packages/trpc/src/services/deltaTestTrigger.ts.

4. Job payload + worker

• Confirm TestRunJob shape can carry the framework discriminator without ambiguity. If it can't, add a framework: 'playwright' | 'vitest' | ... field and gate the executor selection on it.
• Update the worker that consumes the BullMQ queue to select the right executor by framework.

5. Tests

• Unit tests for the new executor's plan→args translation.
• Unit tests for the new framework's spec-selection.
• An integration test that proves the queue → worker → executor → result loop for at least one happy path.

6. Docs

• Update the support matrix above.
• Add framework-specific notes if there are gotchas (Vitest's globals config, Jest's testEnvironment, etc.).

Customer-facing behaviour today

A project without a supported framework:

• Skips the izri/tests child signal entirely. The umbrella Check Run surfaces the other signals (scope, hallucination, visual) normally.
• Does not block the PR on missing tests — there's nothing to run.
• Surfaces in the dashboard as "Tests: not configured" on the delta report.

This is intentional: the project is still useful for scope and hallucination analysis even without an executable test suite.