# Runner deployment (Railway)

> Provisioning guide for the `apps/runner` BullMQ worker. Read this before
> creating the runner service in a new Railway project — the runner's
> configuration shape diverges from the API/web/ai-service services in a
> few places that bite if you treat it as just another worker.

## What the runner is

`apps/runner` is a long-running Node process that consumes test_run jobs
from `izri:test-runs:queue` (BullMQ on the shared Redis), clones the
target repo into a tmpdir, runs the project's `.izri/izri.yml` test
command, and writes results back via the **tRPC API** — not direct DB
access. See `apps/runner/src/reporter.ts` for why (customer-self-hosted
runners can't have `DATABASE_URL`).

It does not serve HTTP. There is no `/health` endpoint; Railway falls
back to process-liveness which is the right signal for a worker.

## Prerequisites

You need an existing Railway project with three services already up:

| Service | Provides |
| ------- | -------- |
| `api`   | tRPC endpoint the runner reports back to (`API_URL`). |
| Postgres plugin | Backs the API. The runner never touches it directly. |
| Redis plugin | Job queue + event bus (`REDIS_URL`). |

If those aren't there, work through `deploy/README.md` first; this doc
assumes they exist.

## Step 1: mint the runner API token

The runner authenticates to the API with a long-lived bearer token
scoped to `runner:write`. Mint it once from a workstation that has
`DATABASE_URL` for the **production** database:

```bash
DATABASE_URL=postgresql://... \
  pnpm --filter @izri/database mint:runner-token
```

The script (see `packages/database/scripts/mint-runner-token.ts`) is
idempotent — it provisions a stable `izri-platform` org + system user
and prints the new token. **Copy the `izri_…` value immediately**, it
is shown only once.

Bound-org note: the runner scope grants cross-org access (`runner.ts`
filters by `runner:write` scope, not org). The token is *bound* to the
internal `izri-platform` org so deleting an end-user org never bricks
the runner.

## Step 2: create the Railway service

In the Railway dashboard:

1. **Create service** → Empty Service → name it `runner`.
2. **Settings → Source** → connect the `legendify-dev/izri` repo + the
   branch you want to track (typically `main`).
3. **Settings → Config-as-code** → set the path to
   `deploy/runner/railway.toml`.
4. **Settings → Root Directory** → leave **empty**. The Dockerfile uses
   the repo root as the build context (pnpm workspaces need it).
5. **Settings → Variables** → set:

   | Variable           | Value |
   | ------------------ | ----- |
   | `API_URL`          | the public URL of the API service in this project |
   | `REDIS_URL`        | reference the Redis plugin's `${{REDIS_URL}}` |
   | `RUNNER_API_TOKEN` | the `izri_…` you minted in step 1 |
   | `NODE_ENV`         | `production` |

   Optional / for visual snapshots (epic #227):

   | Variable             | Value |
   | -------------------- | ----- |
   | `R2_ENDPOINT`        | same as API |
   | `R2_ACCESS_KEY_ID`   | same as API |
   | `R2_SECRET_ACCESS_KEY` | same as API |
   | `R2_BUCKET`          | same as API |
   | `R2_PUBLIC_URL_BASE` | same as API |

   See `env.config.ts → services.runner` for the canonical list. Anything
   not listed there is unused by the runner image.

6. **Deploy.** Railway picks up `deploy/runner/railway.toml`, which uses
   `docker/runner/Dockerfile` as the build, restarts on failure (10
   max), and runs one replica.

## Step 3: verify the runner is consuming

```bash
railway logs --service runner
```

Expected lines:

```
runner started — waiting for jobs
reaper: marked 0 stuck RUNNING rows as ERROR
```

The reaper line repeats every BRPOP cycle (~30s). If you only see the
`runner started` line and no reaper output, the runner is alive but
unable to call back to the API — most likely a wrong `API_URL` or a
mistyped `RUNNER_API_TOKEN`. Both surface as `HTTP 401 Authentication
required` in the runner logs.

The API side runs a [presence watchdog](../../packages/trpc/src/services/runnerPresenceWatchdog.ts)
(#259) that logs a WARN once the queue stays non-empty for >60s. If you
see that warning on the API and not corresponding runner activity, the
runner's BRPOP isn't seeing the jobs — typically a `REDIS_URL` mismatch
between the api and runner services.

## Step 4 (optional): Docker daemon caveat

The runner uses [testcontainers](https://node.testcontainers.org/) to
spin up declared services (Postgres, Redis, …) from the target repo's
`.izri/izri.yml`. testcontainers needs a reachable Docker daemon, and
Railway's container runtime doesn't expose one (privileged containers
disallowed, no host socket, rootless dockerd blocked at the
user-namespace level — see `deploy/README.md → "Staging runner: runs
on the laptop"` for the full investigation).

**Implication**: a Railway-hosted runner is fine for repos that don't
declare `services:` in their `.izri/izri.yml`. For repos that do
(including this monorepo's dogfood config), the runner needs a
Docker-capable host: a Fly Machine, a small VPS, or Testcontainers
Cloud. Track that work in #271.

Until then, the runner service can be **paused** in the Railway
dashboard — `pnpm dev:runner` on a developer laptop drains the queue.
See `deploy/README.md`.

## Re-provisioning checklist (other environments)

To stand up the same shape in a sibling Railway project (staging vs
prod, demo environments, etc.) walk this checklist:

- [ ] api + postgres + redis services are healthy in the target project.
- [ ] `RUNNER_API_TOKEN` minted against the target project's DB.
- [ ] Service created with `deploy/runner/railway.toml` config-as-code.
- [ ] All env vars set per Step 2 table.
- [ ] Logs show `runner started — waiting for jobs`.
- [ ] Trigger a manual run from the dashboard; confirm it advances past
      `PENDING` within ~30s and reaches a terminal state.

If you encounter `RUNNER_API_TOKEN not set — mint one with …`, the
service variable is missing. If you see `HTTP 401`, the token is set
but the API doesn't recognize it (wrong DB, wrong scope, or rotation
happened on one side).