Runner deployment (Railway)

Provisioning guide for the apps/runner BullMQ worker. Read this before creating the runner service in a new Railway project — the runner's configuration shape diverges from the API/web/ai-service services in a few places that bite if you treat it as just another worker.

What the runner is

apps/runner is a long-running Node process that consumes test_run jobs from izri:test-runs:queue (BullMQ on the shared Redis), clones the target repo into a tmpdir, runs the project's .izri/izri.yml test command, and writes results back via the tRPC API — not direct DB access. See apps/runner/src/reporter.ts for why (customer-self-hosted runners can't have DATABASE_URL).

It does not serve HTTP. There is no /health endpoint; Railway falls back to process-liveness which is the right signal for a worker.

Prerequisites

You need an existing Railway project with three services already up:

Service	Provides
`api`	tRPC endpoint the runner reports back to (`API_URL`).
Postgres plugin	Backs the API. The runner never touches it directly.
Redis plugin	Job queue + event bus (`REDIS_URL`).

If those aren't there, work through deploy/README.md first; this doc assumes they exist.

Step 1: mint the runner API token

The runner authenticates to the API with a long-lived bearer token scoped to runner:write. Mint it once from a workstation that has DATABASE_URL for the production database:

DATABASE_URL=postgresql://... \
  pnpm --filter @izri/database mint:runner-token

The script (see packages/database/scripts/mint-runner-token.ts) is idempotent — it provisions a stable izri-platform org + system user and prints the new token. Copy the izri_… value immediately, it is shown only once.

Bound-org note: the runner scope grants cross-org access (runner.ts filters by runner:write scope, not org). The token is bound to the internal izri-platform org so deleting an end-user org never bricks the runner.

Step 2: create the Railway service

In the Railway dashboard:

Create service → Empty Service → name it runner.
Settings → Source → connect the legendify-dev/izri repo + the branch you want to track (typically main).
Settings → Config-as-code → set the path to deploy/runner/railway.toml.
Settings → Root Directory → leave empty. The Dockerfile uses the repo root as the build context (pnpm workspaces need it).

Settings → Variables → set:

Variable	Value
`API_URL`	the public URL of the API service in this project
`REDIS_URL`	reference the Redis plugin's `${{REDIS_URL}}`
`RUNNER_API_TOKEN`	the `izri_…` you minted in step 1
`NODE_ENV`	`production`

Optional / for visual snapshots (epic #227):

Variable	Value
`R2_ENDPOINT`	same as API
`R2_ACCESS_KEY_ID`	same as API
`R2_SECRET_ACCESS_KEY`	same as API
`R2_BUCKET`	same as API
`R2_PUBLIC_URL_BASE`	same as API

See env.config.ts → services.runner for the canonical list. Anything not listed there is unused by the runner image.

Deploy. Railway picks up deploy/runner/railway.toml, which uses docker/runner/Dockerfile as the build, restarts on failure (10 max), and runs one replica.

Step 3: verify the runner is consuming

railway logs --service runner

Expected lines:

runner started — waiting for jobs
reaper: marked 0 stuck RUNNING rows as ERROR

The reaper line repeats every BRPOP cycle (~30s). If you only see the runner started line and no reaper output, the runner is alive but unable to call back to the API — most likely a wrong API_URL or a mistyped RUNNER_API_TOKEN. Both surface as HTTP 401 Authentication required in the runner logs.

The API side runs a presence watchdog (#259) that logs a WARN once the queue stays non-empty for >60s. If you see that warning on the API and not corresponding runner activity, the runner's BRPOP isn't seeing the jobs — typically a REDIS_URL mismatch between the api and runner services.

Step 4 (optional): Docker daemon caveat

The runner uses testcontainers to spin up declared services (Postgres, Redis, …) from the target repo's .izri/izri.yml. testcontainers needs a reachable Docker daemon, and Railway's container runtime doesn't expose one (privileged containers disallowed, no host socket, rootless dockerd blocked at the user-namespace level — see deploy/README.md → "Staging runner: runs on the laptop" for the full investigation).

Implication: a Railway-hosted runner is fine for repos that don't declare services: in their .izri/izri.yml. For repos that do (including this monorepo's dogfood config), the runner needs a Docker-capable host: a Fly Machine, a small VPS, or Testcontainers Cloud. Track that work in #271.

Until then, the runner service can be paused in the Railway dashboard — pnpm dev:runner on a developer laptop drains the queue. See deploy/README.md.

Re-provisioning checklist (other environments)

To stand up the same shape in a sibling Railway project (staging vs prod, demo environments, etc.) walk this checklist:

api + postgres + redis services are healthy in the target project.
RUNNER_API_TOKEN minted against the target project's DB.
Service created with deploy/runner/railway.toml config-as-code.
All env vars set per Step 2 table.
Logs show runner started — waiting for jobs.
Trigger a manual run from the dashboard; confirm it advances past PENDING within ~30s and reaches a terminal state.

If you encounter RUNNER_API_TOKEN not set — mint one with …, the service variable is missing. If you see HTTP 401, the token is set but the API doesn't recognize it (wrong DB, wrong scope, or rotation happened on one side).