Troubleshooting Guide

Common issues and fixes for the E2E testing platform.

Docker Not Running

Symptom: Test runs stay in PENDING or immediately go to ERROR with a message like Cannot connect to the Docker daemon.

Fix:

# Check Docker is running
docker info

# Start Docker Desktop (macOS/Windows) or:
sudo systemctl start docker  # Linux

# Verify the Playwright image is accessible
docker run --rm mcr.microsoft.com/playwright:v1.42.0-jammy npx playwright --version

Notes:

• The Node.js API process must have access to the Docker socket (/var/run/docker.sock)
• In containerized deployments, you may need to mount the socket: -v /var/run/docker.sock:/var/run/docker.sock
• The default test run timeout is 5 minutes (300,000ms). Override with executorOptions.timeoutMs.

API Key Missing / AI Service Returns Mock Data

Symptom: /suggest-tests returns placeholder text or [MOCK RESPONSE] in results. GET /provider returns "mock".

Fix:

# Check current provider
curl http://localhost:8000/provider

# Set an API key
export OPENAI_API_KEY=sk-...
# or
export ANTHROPIC_API_KEY=sk-ant-...

# Restart the AI service
uvicorn main:app --host 0.0.0.0 --port 8000

Notes:

• Setting AI_PROVIDER=mock forces mock responses regardless of API keys
• The Node.js tRPC layer calls the AI service at AI_SERVICE_URL (default http://localhost:8000). If the AI service is down, analysis.suggestTests will throw a 500.
• The AI service must be running separately — it's not part of the main Node.js process.

Webhook Secret Wrong / Events Rejected with 400

Symptom: GitHub shows webhook deliveries with a 400 response. Logs show "Signature mismatch" or "Missing X-Hub-Signature-256 header".

Causes:

1. Secret in GitHub doesn't match what's stored in the database
2. The webhook URL has the wrong projectId
3. GitHub is not sending the signature header (content type must be application/json)

Fix:

# 1. Rotate the secret via tRPC:
#    webhooks.rotateSecret({ projectId: "..." })
# 2. Update the secret in GitHub: Settings → Webhooks → Edit
# 3. Ensure Content type is "application/json" (not "application/x-www-form-urlencoded")

Debugging: Check webhooks.listEvents for recent events — the error field shows what went wrong.

Rate Limits (429 Too Many Requests)

Symptom: API calls fail with TOO_MANY_REQUESTS or HTTP 429. Error message: Rate limit exceeded. Max X requests per Ys.

Limits:

Fix:

• Wait for the window to reset (1 minute)
• In development/testing, the rate limiter store can be cleared by restarting the API
• Rate limits are in-memory — they reset on server restart and are not shared across instances

Tests Not Triggering on Push

Symptom: Webhook is delivered (GitHub shows 200) but no test run is created.

Check:

1. Branch filter — webhooks.getConfig → is the push branch in branches? (empty list = all branches)
2. File pattern — is at least one changed file matching filePatterns? Use a glob tester. (empty = any file)
3. Deduplication — was this exact commitSha already run within the last hour? Check testRuns.list for the project.
4. Debounce — if debounce is enabled, the test run is delayed. The webhook event logs show "Debounced: waiting Xms".

Test Run Stuck in PENDING

Symptom: Test run was created but never moves to PASSED/FAILED/ERROR.

Possible causes:

• Docker container timed out silently (check error field after timeout)
• The executeTestRun call was never awaited (fire-and-forget pattern issue)
• Docker image pull is slow on first run

Fix:

# Check if any containers are stuck
docker ps -a | grep playwright

# Kill stuck containers
docker rm -f <container-id>

Database Connection Issues

Symptom: API fails to start with connection refused or ECONNREFUSED on port 5432.

Fix:

# Check PostgreSQL is running
pg_isready -h localhost -p 5432

# Verify DATABASE_URL is correct
echo $DATABASE_URL

# Re-run migrations
pnpm db:migrate

AI Service Connection Refused

Symptom: analysis.suggestTests throws fetch failed or ECONNREFUSED :8000.

Fix:

# Check AI service is running
curl http://localhost:8000/health

# If using a non-default URL, ensure AI_SERVICE_URL is set in the API environment
export AI_SERVICE_URL=http://your-ai-service:8000

GitHub Commit Status Not Appearing

Symptom: Tests run fine but no status check appears on the commit in GitHub.

Fix:

• GitHub commit status posting requires a token to be configured in your project settings (not as a server environment variable — the token is passed programmatically to postGitHubStatus())
• The token needs repo:status scope (or statuses:write for fine-grained tokens)
• Check the repoFullName is correct (format: owner/repo)
• Status posting is best-effort — failures are silently swallowed. Check API logs for errors.

izri/quality Check Run Stuck at "in_progress"

Symptom: The umbrella izri/quality Check Run on a PR never reaches a terminal state — it sits at "in progress" even though the per-signal children (izri/scope, izri/tests, izri/hallucination, izri/visual) have all completed. The sticky PR comment shows "Analysis in progress" indefinitely.

Cause: One of the child analyzers errored without producing a hard finding. The umbrella aggregator (packages/trpc/src/services/deltaReportAggregation.ts) classifies the delta_report as status='error' with umbrellaState='unknown' because at least one child is in error state. Up to PR #291 the surface adapters mapped any non-complete status to in_progress, so the rollup never finalized.

Fix: PR #291 lands the rollup-terminalization fix:

• (error, failing) — analyzer error with a hard finding (e.g. runner ERROR produces a runner_error hard finding) → umbrella finalizes as failure / "Delta needs attention".
• (error, unknown) with at least one sibling child still pending — umbrella stays in_progress so the sibling's completion can refine the verdict (race-safe retry path).
• (error, unknown) with all children terminal — falls back to neutral / "Delta has open questions" to avoid an indefinite spinner.

Existing stuck rows: PRs opened before #291 deployed to staging stay stuck — the fix is forward-looking. To unstick a specific row, re-emit test_run.completed for its test_run_id (the BullMQ dispatcher runs recomputeDeltaReport + the surface adapters, which now use the new terminal logic). See issue #264 for the planned backfill helper.

Related: issue #292 tracks the follow-up that makes hallucination/visual/scope analyzer errors produce hard findings symmetrically with runner_error, so the rollup lands at failure instead of neutral when those analyzers crash.