# Test framework support

> Resolution of [#267](https://github.com/legendify-dev/izri/issues/267).
> The framework gate is a code-level constant by design — this doc explains
> why and what to do when adding a new framework.

## Current support matrix

| Framework  | Spec discovery     | Spec execution                                | Tier      | Status      |
| ---------- | ------------------ | --------------------------------------------- | --------- | ----------- |
| Playwright | `playwright test`  | Docker runner via `@izri/test-executor`       | All tiers | **Shipped** |
| Vitest     | —                  | —                                             | —         | Not planned |
| Jest       | —                  | —                                             | —         | Not planned |
| Cypress    | —                  | —                                             | —         | Not planned |

Only Playwright is supported today. The other entries are placeholders — see
[Adding a framework](#adding-a-framework) below for what shipping each would
involve.

## How the gate works

A project participates in the `izri/tests` signal only if it declares a
supported framework on its `projects.test_frameworks` column. That column is
populated by the repo-analyzer (`analyzeRepository`) and can be backfilled
on existing projects via the bulk-backfill admin procedure (see
[#266](https://github.com/legendify-dev/izri/issues/266) /
[`analysis.backfillTestFrameworks`](../api-reference/trpc-routers.md)).

The gate itself is one constant in
[`packages/trpc/src/services/deltaTestTrigger.ts`](../../packages/trpc/src/services/deltaTestTrigger.ts):

```ts
const SUPPORTED_FRAMEWORKS = new Set(['playwright'])
```

`triggerTestRunForDelta` checks each delta's project against this set and
returns `{ triggered: false, reason: 'no_supported_framework' }` for anything
that doesn't match. Frameworks are compared case-insensitively.

## Why code-level rather than per-project setting

The expansion path discussed in PR #211 considered moving the gate to a
per-project setting backed by a framework registry. We landed on code-level
for the following reasons:

1. **Adding a framework is always a code change anyway.** The runner
   (`@izri/test-executor`) needs to know how to discover specs, install
   dependencies, and invoke the framework binary. That's per-framework code,
   not configuration.
2. **No customer is currently asking for it.** 100% of shipped customers are
   on Playwright. Building a runtime registry now would be designing against
   zero data points.
3. **The constant is auditable.** Every PR that wants to expand framework
   support touches one obvious file. Reviewers can spot it. A registry that
   "just works" when admins flip a feature flag is much harder to reason
   about.
4. **Migration is cheap.** When we have two or three frameworks shipped, we
   can convert the constant to a registry without breaking anything — the
   wire format on the project row already allows multiple values.

Revisit this decision when:

- A paying customer needs a non-Playwright framework on a deadline that
  the code-level expansion can't meet.
- The runner gains the ability to install + execute a framework dynamically
  (today every framework requires bespoke runner code).

## Adding a framework

Expansion checklist when a new framework lands. Treat each step as gated —
don't merge a half-built path.

### 1. Runner support (`@izri/test-executor`)

- Add an executor implementation in `packages/test-executor/src/`. It must
  match the existing `DockerPlaywrightExecutor` shape: dispose, run, parse
  results into the canonical `TestRunResult`.
- Add framework detection in `packages/repo-analyzer/` so
  `analyzeRepository` populates `projects.test_frameworks` correctly for
  the new framework.
- Choose the docker base image and the install/exec commands. Document any
  framework-specific environment assumptions (Node version, etc.).

### 2. Spec selector (`specSelector`)

- Update `packages/trpc/src/services/specSelector.ts` (or its sibling test
  selector logic) to understand the framework's spec-discovery rules.
  Playwright treats positional args as file globs; Vitest does similar but
  with different defaults; Jest needs `--testPathPattern`; etc.
- Update `TestSelectionPlan` to carry framework-aware arguments if the new
  framework needs flags Playwright doesn't (e.g. `--config` paths).

### 3. The framework gate

- Add the framework name (lowercase) to `SUPPORTED_FRAMEWORKS` in
  `packages/trpc/src/services/deltaTestTrigger.ts`.

### 4. Job payload + worker

- Confirm `TestRunJob` shape can carry the framework discriminator without
  ambiguity. If it can't, add a `framework: 'playwright' | 'vitest' | ...`
  field and gate the executor selection on it.
- Update the worker that consumes the BullMQ queue to select the right
  executor by framework.

### 5. Tests

- Unit tests for the new executor's plan→args translation.
- Unit tests for the new framework's spec-selection.
- An integration test that proves the queue → worker → executor → result
  loop for at least one happy path.

### 6. Docs

- Update the support matrix above.
- Add framework-specific notes if there are gotchas (Vitest's `globals`
  config, Jest's `testEnvironment`, etc.).

## Customer-facing behaviour today

A project without a supported framework:

- Skips the `izri/tests` child signal entirely. The umbrella Check Run
  surfaces the other signals (scope, hallucination, visual) normally.
- Does not block the PR on missing tests — there's nothing to run.
- Surfaces in the dashboard as *"Tests: not configured"* on the delta
  report.

This is intentional: the project is still useful for scope and hallucination
analysis even without an executable test suite.