Architecture: E2E Testing MVP

System overview for the automated E2E testing pipeline.

System Diagram

┌─────────────────────────────────────────────────────────────────────┐
│  Developer                                                           │
│  git push → GitHub                                                   │
└────────────────────────────┬────────────────────────────────────────┘
                             │ POST /api/webhooks/github/:projectId
                             │ (X-Hub-Signature-256 header)
                             ▼
┌─────────────────────────────────────────────────────────────────────┐
│  apps/api  (Hono + tRPC)                                            │
│                                                                      │
│  Webhook HTTP Route                                                  │
│    ├─ verifyGitHubSignature()  [HMAC-SHA256, timing-safe]           │
│    ├─ rateLimiter (30 req/min)                                       │
│    └─ webhookService.processEvent()                                  │
│         │                                                            │
│         ▼                                                            │
│  testTriggerService                                                  │
│    ├─ shouldTriggerAnalysis()  [branch + file pattern filter]       │
│    ├─ dedup check  [commitSha × project × eventType, 1h window]     │
│    ├─ debounce  [coalesce rapid pushes]                              │
│    ├─ db.insert(testRuns, { status: 'PENDING' })                    │
│    ├─ postGitHubStatus('pending')                                    │
│    └─ executeTestRun()                                               │
│         │                                                            │
│         ▼                                                            │
│  DockerPlaywrightExecutor  (@izri/test-executor)               │
│    ├─ Pulls mcr.microsoft.com/playwright:v1.42.0-jammy              │
│    ├─ Mounts testDir → /tests (read-only)                           │
│    ├─ Runs: npx playwright test                                      │
│    └─ Returns { exitCode, stdout, stderr, passed, failed, errors }  │
│         │                                                            │
│         ▼                                                            │
│  ResultCollector + ReportFormatter                                   │
│    ├─ Parses Playwright output                                       │
│    ├─ Builds structured TestReport                                   │
│    └─ Formats plain-text summary for logs                           │
│         │                                                            │
│         ▼                                                            │
│  reportTestRunResult()                                               │
│    ├─ db.update(testRuns, { status, passedTests, results, ... })    │
│    └─ postGitHubStatus('success' | 'failure' | 'error')            │
└─────────────────────────────┬───────────────────────────────────────┘
                              │
           ┌──────────────────┴──────────────────┐
           │                                      │
           ▼                                      ▼
  ┌──────────────────┐               ┌───────────────────────────┐
  │  PostgreSQL       │               │  Docker daemon            │
  │                  │               │  (test container)          │
  │  test_runs       │               │                           │
  │  webhook_events  │               │  Playwright tests run     │
  │  webhook_configs │               │  in isolation             │
  └──────────────────┘               └───────────────────────────┘

  Optional: AI Test Suggestions
  ────────────────────────────
  tRPC: analysis.suggestTests
    │
    ├─ repo-analyzer  → AnalysisResult (code structure, coverage, deps)
    │
    └─ POST http://ai-service:8000/suggest-tests
         │
         ▼
    ┌─────────────────────────────────────┐
    │  AI Service  (FastAPI / Python)     │
    │                                     │
    │  RepoAnalyzerIntegration            │
    │    └─ extracts context from         │
    │       AnalysisResult                │
    │                                     │
    │  AI Provider (auto-detected)        │
    │    ├─ OpenAI (OPENAI_API_KEY)       │
    │    ├─ Anthropic (ANTHROPIC_API_KEY) │
    │    └─ Mock (no key)                 │
    │                                     │
    │  Returns: TestSuggestionResponse    │
    │    - commit_sha                     │
    │    - summary                        │
    │    - suggestions[]                  │
    └─────────────────────────────────────┘

Component Overview

apps/api — Main HTTP API

• Framework: Hono (HTTP) + tRPC (typed procedures)
• Responsibilities: Auth, webhook ingestion, project management, test run orchestration
• Key routers: webhooks, testTriggers, projects, tokens
• Key middleware: rateLimiter (in-memory, per-IP/user, configurable window)

packages/trpc — tRPC Routers & Services

• webhookService: Validates signatures, stores events, dispatches to trigger service
• testTriggerService: Evaluates trigger conditions, manages debounce/dedup, creates test run records, calls DockerPlaywrightExecutor, reports results
• aiServiceClient: Typed HTTP client for the Python AI service
• rateLimiter: tRPC middleware factory — createRateLimiter({ max, windowMs })

packages/test-executor — Docker Test Execution

• DockerPlaywrightExecutor: Runs Playwright in the official Docker image
• dockerRunner: Low-level Docker container lifecycle (create, start, wait, logs, remove)
• outputParser: Parses Playwright plain-text output into { passed, failed, errors }
• ResultCollector: Builds a structured TestReport from PlaywrightTestResult
• ReportFormatter: Renders TestReport as plain text or JSON

packages/ai-service — Python AI Service (FastAPI)

• Endpoints: GET /health, GET /provider, POST /suggest-tests, POST /generate-tests, POST /analyze
• Providers: OpenAI (openai SDK), Anthropic (anthropic SDK), Mock (no key needed)
• RepoAnalyzerIntegration: Extracts changed files, detected languages, and coverage info from AnalysisResult and builds an AI prompt

packages/repo-analyzer — Repository Analysis

• Scans codebase: code structure, dependencies, staging environment, configuration
• Produces AnalysisResult consumed by the AI service

packages/database — Database Schema (Drizzle ORM)

• test_runs: Records per test execution — commitSha, executorType, status, results (JSONB), dedup index on commitSha
• webhook_configs: Per-project webhook settings (branches, filePatterns, secret hash, active flag)
• webhook_events: Audit log of every received webhook delivery with retry state

Data Flow: Webhook → Test Result

1. GitHub push event → POST /api/webhooks/github/:projectId
2. Signature verified via HMAC-SHA256
3. Event stored in webhook_events (status: PENDING)
4. shouldTriggerAnalysis() checks branch + file filters
5. Dedup check: query test_runs for same commitSha in last hour
6. Debounce: optional delay to coalesce rapid pushes
7. test_runs row inserted (status: PENDING, executorType: docker)
8. GitHub commit status: pending
9. DockerPlaywrightExecutor.run() → Docker container runs tests
10. Output parsed → ResultCollector.collect() → TestReport
11. test_runs updated with status + results
12. GitHub commit status: success / failure / error

Rate Limiting

Rate limiting is implemented as tRPC middleware using an in-memory store (Map).

• Default: 60 req/min per authenticated user ID (falls back to IP)
• Webhook procedures: 30 req/min per IP
• Each createRateLimiter() call creates an isolated store
• Store resets on server restart; not shared across multiple API instances
• Returns TRPCError({ code: 'TOO_MANY_REQUESTS' }) (HTTP 429) when exceeded

Key Design Decisions

• Docker isolation: Tests run in a locked-down container with the test directory mounted read-only, preventing test code from affecting the host
• Timing-safe signature verification: Uses crypto.subtle with constant-time comparison to prevent timing attacks on webhook secrets
• Dedup by commitSha: Prevents duplicate runs when the same commit triggers multiple events (e.g., PR sync + push)
• AI provider auto-detection: The AI service works out-of-the-box without any API key configured (uses mock), making local development easy
• Best-effort GitHub status: Status posting failures are silently caught — a GitHub API error won't fail the test run