# Troubleshooting Guide

Common issues and fixes for the E2E testing platform.

---

## Docker Not Running

**Symptom**: Test runs stay in `PENDING` or immediately go to `ERROR` with a message like `Cannot connect to the Docker daemon`.

**Fix**:
```bash
# Check Docker is running
docker info

# Start Docker Desktop (macOS/Windows) or:
sudo systemctl start docker  # Linux

# Verify the Playwright image is accessible
docker run --rm mcr.microsoft.com/playwright:v1.42.0-jammy npx playwright --version
```

**Notes**:
- The Node.js API process must have access to the Docker socket (`/var/run/docker.sock`)
- In containerized deployments, you may need to mount the socket: `-v /var/run/docker.sock:/var/run/docker.sock`
- The default test run timeout is 5 minutes (300,000ms). Override with `executorOptions.timeoutMs`.

---

## API Key Missing / AI Service Returns Mock Data

**Symptom**: `/suggest-tests` returns placeholder text or `[MOCK RESPONSE]` in results. `GET /provider` returns `"mock"`.

**Fix**:
```bash
# Check current provider
curl http://localhost:8000/provider

# Set an API key
export OPENAI_API_KEY=sk-...
# or
export ANTHROPIC_API_KEY=sk-ant-...

# Restart the AI service
uvicorn main:app --host 0.0.0.0 --port 8000
```

**Notes**:
- Setting `AI_PROVIDER=mock` forces mock responses regardless of API keys
- The Node.js tRPC layer calls the AI service at `AI_SERVICE_URL` (default `http://localhost:8000`). If the AI service is down, `analysis.suggestTests` will throw a 500.
- The AI service must be running separately — it's not part of the main Node.js process.

---

## Webhook Secret Wrong / Events Rejected with 400

**Symptom**: GitHub shows webhook deliveries with a 400 response. Logs show "Signature mismatch" or "Missing X-Hub-Signature-256 header".

**Causes**:
1. Secret in GitHub doesn't match what's stored in the database
2. The webhook URL has the wrong `projectId`
3. GitHub is not sending the signature header (content type must be `application/json`)

**Fix**:
```bash
# 1. Rotate the secret via tRPC:
#    webhooks.rotateSecret({ projectId: "..." })
# 2. Update the secret in GitHub: Settings → Webhooks → Edit
# 3. Ensure Content type is "application/json" (not "application/x-www-form-urlencoded")
```

**Debugging**: Check `webhooks.listEvents` for recent events — the `error` field shows what went wrong.

---

## Rate Limits (429 Too Many Requests)

**Symptom**: API calls fail with `TOO_MANY_REQUESTS` or HTTP 429. Error message: `Rate limit exceeded. Max X requests per Ys.`

**Limits**:
| Endpoint | Limit |
|----------|-------|
| Default API procedures | 60 req/min per IP/user |
| Webhook procedures (`webhooks.configure`, etc.) | 30 req/min per IP |
| Incoming webhook HTTP endpoint | 30 req/min per IP |

**Fix**:
- Wait for the window to reset (1 minute)
- In development/testing, the rate limiter store can be cleared by restarting the API
- Rate limits are in-memory — they reset on server restart and are not shared across instances

---

## Tests Not Triggering on Push

**Symptom**: Webhook is delivered (GitHub shows 200) but no test run is created.

**Check**:
1. **Branch filter** — `webhooks.getConfig` → is the push branch in `branches`? (empty list = all branches)
2. **File pattern** — is at least one changed file matching `filePatterns`? Use a glob tester. (empty = any file)
3. **Deduplication** — was this exact `commitSha` already run within the last hour? Check `testRuns.list` for the project.
4. **Debounce** — if debounce is enabled, the test run is delayed. The webhook event logs show `"Debounced: waiting Xms"`.

---

## Test Run Stuck in PENDING

**Symptom**: Test run was created but never moves to PASSED/FAILED/ERROR.

**Possible causes**:
- Docker container timed out silently (check `error` field after timeout)
- The `executeTestRun` call was never awaited (fire-and-forget pattern issue)
- Docker image pull is slow on first run

**Fix**:
```bash
# Check if any containers are stuck
docker ps -a | grep playwright

# Kill stuck containers
docker rm -f <container-id>
```

---

## Database Connection Issues

**Symptom**: API fails to start with `connection refused` or `ECONNREFUSED` on port 5432.

**Fix**:
```bash
# Check PostgreSQL is running
pg_isready -h localhost -p 5432

# Verify DATABASE_URL is correct
echo $DATABASE_URL

# Re-run migrations
pnpm db:migrate
```

---

## AI Service Connection Refused

**Symptom**: `analysis.suggestTests` throws `fetch failed` or `ECONNREFUSED :8000`.

**Fix**:
```bash
# Check AI service is running
curl http://localhost:8000/health

# If using a non-default URL, ensure AI_SERVICE_URL is set in the API environment
export AI_SERVICE_URL=http://your-ai-service:8000
```

---

## GitHub Commit Status Not Appearing

**Symptom**: Tests run fine but no status check appears on the commit in GitHub.

**Fix**:
- GitHub commit status posting requires a token to be configured in your project settings (not as a server environment variable — the token is passed programmatically to `postGitHubStatus()`)
- The token needs `repo:status` scope (or `statuses:write` for fine-grained tokens)
- Check the `repoFullName` is correct (format: `owner/repo`)
- Status posting is best-effort — failures are silently swallowed. Check API logs for errors.

---

## `izri/quality` Check Run Stuck at "in_progress"

**Symptom**: The umbrella `izri/quality` Check Run on a PR never reaches a terminal state — it sits at "in progress" even though the per-signal children (`izri/scope`, `izri/tests`, `izri/hallucination`, `izri/visual`) have all completed. The sticky PR comment shows _"Analysis in progress"_ indefinitely.

**Cause**: One of the child analyzers errored without producing a hard finding. The umbrella aggregator (`packages/trpc/src/services/deltaReportAggregation.ts`) classifies the delta_report as `status='error'` with `umbrellaState='unknown'` because at least one child is in error state. Up to PR #291 the surface adapters mapped any non-`complete` status to `in_progress`, so the rollup never finalized.

**Fix**: PR #291 lands the rollup-terminalization fix:

- `(error, failing)` — analyzer error with a hard finding (e.g. runner `ERROR` produces a `runner_error` hard finding) → umbrella finalizes as `failure / "Delta needs attention"`.
- `(error, unknown)` with at least one sibling child still pending — umbrella stays `in_progress` so the sibling's completion can refine the verdict (race-safe retry path).
- `(error, unknown)` with all children terminal — falls back to `neutral / "Delta has open questions"` to avoid an indefinite spinner.

**Existing stuck rows**: PRs opened before #291 deployed to staging stay stuck — the fix is forward-looking. To unstick a specific row, re-emit `test_run.completed` for its `test_run_id` (the BullMQ dispatcher runs `recomputeDeltaReport` + the surface adapters, which now use the new terminal logic). See issue #264 for the planned backfill helper.

**Related**: issue #292 tracks the follow-up that makes hallucination/visual/scope analyzer errors produce hard findings symmetrically with `runner_error`, so the rollup lands at `failure` instead of `neutral` when those analyzers crash.