AI Agent Execution Guide: E2E Testing MVP
Step-by-step implementation guide for AI agents to build the E2E testing MVP
🤖 Agent Context Setup
What You Need to Know
- Project: Izri - Automated E2E testing platform with AI agents
- Goal: Build MVP for automated E2E testing using staging URLs (no repository cloning initially)
- Architecture: Monorepo with TypeScript packages and Python AI service
- Current State: Repository analyzer is production-ready, AI service is mocked, test execution is planned
Key Strategic Decisions
- URL-First Approach: Start with staging URL testing, add repo cloning later
- Leverage Existing Infrastructure: Repository analyzer is 90% complete
- AI Service Integration: Use OpenAI/Anthropic vs building custom models
- Docker-Based Execution: Isolated test execution with Playwright
🏗️ Phase 1 Implementation (Weeks 1-3)
1.1 Repository Analysis Extensions
Step 1.1.1: Create Staging Environment Plugin
Context: Extend existing repo-analyzer to detect staging URLs from configuration files
Commands to Execute:
# Navigate to repo-analyzer package
cd packages/repo-analyzer
# Create staging environment plugin
touch src/analysis/staging-environment-plugin.ts
Implementation:
// File: packages/repo-analyzer/src/analysis/plugins/staging-environment.ts
import { AnalysisPlugin, RepoContext, AnalysisResult } from '../types/intake'
export class StagingEnvironmentPlugin implements AnalysisPlugin {
id = 'staging-environment'
name = 'Staging Environment Detection'
description = 'Detects and configures staging environment URLs for E2E testing'
supportedExtensions = ['.json', '.yaml', '.yml', '.env', '.md']
async analyze(context: RepoContext): Promise<AnalysisResult> {
const stagingUrls = await this.detectStagingUrls(context)
const environments = await this.detectEnvironments(context)
const configurations = await this.detectConfigurations(context)
return {
pluginId: this.id,
metrics: {
stagingUrlsFound: stagingUrls.length,
environmentsDetected: environments.length,
hasStagingConfig: configurations.length > 0
},
issues: this.validateStagingConfig(stagingUrls, configurations),
metadata: {
stagingUrls,
environments,
configurations,
recommendedUrl: this.recommendStagingUrl(stagingUrls)
},
analyzedAt: new Date().toISOString()
}
}
private async detectStagingUrls(context: RepoContext): Promise<Array<{
url: string
source: string
confidence: number
}>> {
const urls: Array<{ url: string; source: string; confidence: number }> = []
// Scan common configuration files
const configFiles = [
'package.json',
'docker-compose.yml',
'docker-compose.yaml',
'.env.example',
'.env.staging',
'README.md',
'staging.md'
]
for (const file of configFiles) {
const fileContent = await this.readFileContent(context, file)
if (fileContent) {
const foundUrls = this.extractUrls(fileContent)
urls.push(...foundUrls.map(url => ({
url,
source: file,
confidence: this.calculateUrlConfidence(url, file)
})))
}
}
return urls.sort((a, b) => b.confidence - a.confidence)
}
private async detectEnvironments(context: RepoContext): Promise<string[]> {
const environments = ['staging', 'dev', 'development', 'test', 'preview']
const detected: string[] = []
for (const env of environments) {
const hasEnvConfig = context.fileIndex.files.some(file =>
file.path.toLowerCase().includes(env) ||
file.name.toLowerCase().includes(env)
)
if (hasEnvConfig) {
detected.push(env)
}
}
return detected
}
private async detectConfigurations(context: RepoContext): Promise<Array<{
type: string
file: string
content: any
}>> {
const configurations = []
const configPatterns = {
'docker-compose': ['docker-compose.yml', 'docker-compose.yaml'],
'environment': ['.env.staging', '.env.test'],
'deployment': ['vercel.json', 'netlify.toml', 'amplify.yml'],
'ci-cd': ['.github/workflows/staging.yml', '.gitlab-ci.yml']
}
for (const [type, files] of Object.entries(configPatterns)) {
for (const file of files) {
const content = await this.readFileContent(context, file)
if (content) {
try {
const parsed = file.endsWith('.json') ? JSON.parse(content) :
file.endsWith('.yml') || file.endsWith('.yaml') ?
this.parseYaml(content) : content
configurations.push({ type, file, content: parsed })
} catch (error) {
// Skip invalid configurations
}
}
}
}
return configurations
}
private extractUrls(content: string): string[] {
const urlPatterns = [
/https?:\/\/[a-zA-Z0-9-]+\.staging\.[a-zA-Z0-9.-]+/g,
/https?:\/\/staging\.[a-zA-Z0-9.-]+/g,
/https?:\/\/[a-zA-Z0-9-]+-staging\.[a-zA-Z0-9.-]+/g,
/https?:\/\/[a-zA-Z0-9.-]+\/staging/g
]
const urls = new Set<string>()
for (const pattern of urlPatterns) {
const matches = content.match(pattern)
if (matches) {
matches.forEach(url => urls.add(url))
}
}
return Array.from(urls)
}
private calculateUrlConfidence(url: string, source: string): number {
let confidence = 0.5
if (url.includes('staging')) confidence += 0.3
if (['package.json', 'docker-compose.yml', '.env.staging'].includes(source)) {
confidence += 0.2
}
return Math.min(confidence, 1.0)
}
private recommendStagingUrl(urls: Array<{ url: string; confidence: number }>): string | null {
if (urls.length === 0) return null
return urls.reduce((best, current) =>
current.confidence > best.confidence ? current : best
).url
}
private validateStagingConfig(urls: any[], configurations: any[]): any[] {
const issues = []
if (urls.length === 0) {
issues.push({
severity: 'warning',
category: 'staging',
message: 'No staging URLs detected. Manual configuration may be required.'
})
}
const hasHttps = urls.every(url => url.url?.startsWith?.('https://'))
if (!hasHttps) {
issues.push({
severity: 'warning',
category: 'security',
message: 'HTTP URLs detected. Consider using HTTPS for staging environments.'
})
}
return issues
}
private async readFileContent(context: RepoContext, filePath: string): Promise<string | null> {
try {
const fullPath = `${context.repoDir}/${filePath}`
// Implementation would read file from repository
return null // Placeholder
} catch {
return null
}
}
private parseYaml(content: string): any {
// Simple YAML parser - in real implementation use js-yaml
return {}
}
}
Verification:
# Build package
pnpm --filter @izri/repo-analyzer build
# Run tests
pnpm --filter @izri/repo-analyzer test
Step 1.1.2: Extend AIReadyContext Interface
Commands:
# Edit types file
code packages/repo-analyzer/src/types/intake.ts
Add to AIReadyContext interface:
export interface AIReadyContext {
// ... existing fields ...
// New staging environment fields
staging: {
urls: Array<{
url: string
source: string
confidence: number
}>
environments: string[]
configurations: Array<{
type: string
file: string
content: any
}>
recommendedUrl?: string
hasStagingConfig: boolean
}
}
Step 1.1.3: Update EnhancedAnalyzer
Commands:
# Edit enhanced analyzer
code packages/repo-analyzer/src/enhanced-analyzer.ts
Add staging plugin registration:
import { StagingEnvironmentPlugin } from './analysis/staging-environment-plugin'
export class EnhancedRepositoryAnalyzer {
constructor(additionalPlugins: AnalysisPlugin[] = []) {
this.intake = new RepositoryIntake()
// Add staging plugin to default plugins
const defaultPlugins = [
new StagingEnvironmentPlugin(), // Add this
// ... existing plugins
]
this.traditionalPipeline = new TraditionalAnalysisPipeline([...defaultPlugins, ...additionalPlugins])
}
// Update prepareAIContext method to include staging information
private prepareAIContext(analysis: ComprehensiveAnalysis): AIReadyContext {
// ... existing code ...
const stagingResult = results.find((r) => r.pluginId === 'staging-environment')
return {
// ... existing fields ...
staging: {
urls: stagingResult?.metadata.stagingUrls || [],
environments: stagingResult?.metadata.environments || [],
configurations: stagingResult?.metadata.configurations || [],
recommendedUrl: stagingResult?.metadata.recommendedUrl,
hasStagingConfig: stagingResult?.metrics.hasStagingConfig || false
}
}
}
}
1.2 AI Service Real Integration
Step 1.2.1: Install AI SDK Dependencies
Commands:
# Navigate to AI service
cd packages/ai-service
# Update requirements.txt
echo "fastapi==0.115.6
uvicorn[standard]==0.34.0
pydantic==2.10.3
openai==1.58.1
anthropic==0.40.0
python-dotenv==1.0.1
httpx==0.28.1
langchain==0.3.0
langchain-openai==0.2.0" > requirements.txt
Step 1.2.2: Implement OpenAI Client
Commands:
# Create AI client directory
mkdir -p src/ai
# Create OpenAI client
touch src/ai/openai_client.py
Implementation:
# File: packages/ai-service/src/ai/openai_client.py
from typing import Dict, Any, List, Optional
import openai
from openai import OpenAI
import json
import logging
logger = logging.getLogger(__name__)
class OpenAIClient:
def __init__(self, api_key: str, model: str = "gpt-4"):
self.client = OpenAI(api_key=api_key)
self.model = model
async def analyze_repository(self, ai_ready_context: Dict[str, Any]) -> Dict[str, Any]:
"""Analyze repository using OpenAI GPT-4"""
prompt = self._build_analysis_prompt(ai_ready_context)
try:
response = await self.client.chat.completions.acreate(
model=self.model,
messages=[
{"role": "system", "content": "You are an expert software testing consultant."},
{"role": "user", "content": prompt}
],
max_tokens=2000,
temperature=0.3
)
analysis_text = response.choices[0].message.content
return self._parse_analysis_response(analysis_text, ai_ready_context)
except Exception as e:
logger.error(f"OpenAI analysis failed: {e}")
raise
async def generate_e2e_tests(self, ai_ready_context: Dict[str, Any], test_type: str = "e2e") -> Dict[str, Any]:
"""Generate E2E tests based on repository analysis"""
prompt = self._build_test_generation_prompt(ai_ready_context, test_type)
try:
response = await self.client.chat.completions.acreate(
model=self.model,
messages=[
{"role": "system", "content": "You are an expert test automation engineer specializing in E2E testing with Playwright."},
{"role": "user", "content": prompt}
],
max_tokens=3000,
temperature=0.2
)
tests_text = response.choices[0].message.content
return self._parse_test_generation_response(tests_text, ai_ready_context)
except Exception as e:
logger.error(f"Test generation failed: {e}")
raise
def _build_analysis_prompt(self, context: Dict[str, Any]) -> str:
"""Build comprehensive analysis prompt"""
repo_info = context.get('repository', {})
project_info = context.get('project', {})
codebase_info = context.get('codebase', {})
testing_info = context.get('testing', {})
staging_info = context.get('staging', {})
prompt = f"""
Analyze this repository for E2E testing strategy:
REPOSITORY INFO:
- URL: {repo_info.get('url', 'Unknown')}
- Branch: {repo_info.get('branch', 'main')}
- Commit: {repo_info.get('commit', 'latest')}
PROJECT DETAILS:
- Type: {project_info.get('type', 'Unknown')}
- Frameworks: {', '.join(project_info.get('frameworks', []))}
- Testing Frameworks: {', '.join(project_info.get('testingFrameworks', []))}
- Build Tools: {', '.join(project_info.get('buildTools', []))}
CODEBASE STRUCTURE:
- Total Files: {codebase_info.get('totalFiles', 0)}
- Languages: {', '.join(codebase_info.get('languages', {}).keys())}
- Entry Points: {', '.join(codebase_info.get('structure', {}).get('entryPoints', []))}
- API Endpoints: {len(codebase_info.get('structure', {}).get('apiEndpoints', []))} found
TESTING STATUS:
- Has Tests: {testing_info.get('hasTests', False)}
- Test Files: {len(testing_info.get('testFiles', []))}
- Test Frameworks: {', '.join(testing_info.get('testFrameworks', []))}
- Test to Source Ratio: {testing_info.get('testToSourceRatio', 0):.2f}
STAGING ENVIRONMENT:
- Has Staging Config: {staging_info.get('hasStagingConfig', False)}
- Recommended URL: {staging_info.get('recommendedUrl', 'None detected')}
- Environments: {', '.join(staging_info.get('environments', []))}
Based on this analysis, provide:
1. E2E testing strategy recommendations
2. Critical user flows to test
3. Technical implementation approach
4. Potential challenges and solutions
5. Test framework recommendations (Playwright vs Cypress)
Respond in JSON format with keys: strategy, criticalFlows, implementation, challenges, recommendations.
"""
return prompt
def _build_test_generation_prompt(self, context: Dict[str, Any], test_type: str) -> str:
"""Build test generation prompt"""
project_info = context.get('project', {})
codebase_info = context.get('codebase', {})
staging_info = context.get('staging', {})
base_url = staging_info.get('recommendedUrl', 'https://staging.example.com')
frameworks = project_info.get('frameworks', [])
prompt = f"""
Generate comprehensive E2E tests for this application:
APPLICATION DETAILS:
- Frameworks: {', '.join(frameworks)}
- Base URL: {base_url}
- Test Type: {test_type}
Generate Playwright tests in TypeScript that cover:
1. User authentication flows
2. Core user journeys
3. Form submissions and validation
4. Navigation and routing
5. Error handling scenarios
For each test, provide:
- Test file name
- Complete Playwright test code
- Test description and what it validates
- Expected test data setup
Focus on:
- Cross-browser compatibility (Chrome, Firefox, Safari)
- Mobile responsiveness
- Accessibility testing
- Performance considerations
Respond in JSON format with structure:
{{
"tests": [
{{
"fileName": "auth/login.spec.ts",
"description": "User login flow validation",
"code": "complete Playwright test code here",
"testData": {{"email": "test@example.com", "password": "test123"}},
"validates": ["login functionality", "error handling", "redirect after login"]
}}
],
"setupRequired": ["test users", "test data", "environment variables"],
"dependencies": ["@playwright/test", "typescript"],
"estimatedExecutionTime": "2-3 minutes"
}}
"""
return prompt
def _parse_analysis_response(self, response_text: str, context: Dict[str, Any]) -> Dict[str, Any]:
"""Parse OpenAI analysis response"""
try:
import re
json_match = re.search(r'\\{.*\\}', response_text, re.DOTALL)
if json_match:
return json.loads(json_match.group())
else:
return {
"strategy": response_text,
"criticalFlows": ["Manual review required"],
"implementation": "Based on analysis",
"challenges": ["Response parsing required"],
"recommendations": ["Review AI response"]
}
except Exception as e:
logger.error(f"Failed to parse analysis response: {e}")
return {"error": "Failed to parse AI response", "raw_response": response_text}
def _parse_test_generation_response(self, response_text: str, context: Dict[str, Any]) -> Dict[str, Any]:
"""Parse test generation response"""
try:
import re
json_match = re.search(r'\\{.*\\}', response_text, re.DOTALL)
if json_match:
return json.loads(json_match.group())
else:
return {
"tests": [],
"setupRequired": [],
"dependencies": [],
"estimatedExecutionTime": "Unknown",
"error": "Failed to parse test generation response"
}
except Exception as e:
logger.error(f"Failed to parse test generation response: {e}")
return {"error": "Failed to parse test response", "raw_response": response_text}
Step 1.2.3: Update FastAPI Application
Commands:
# Edit main FastAPI app
code packages/ai-service/main.py
Replace mock endpoints with real AI integration:
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from typing import Dict, Any, List, Optional
import os
import logging
from src.ai.openai_client import OpenAIClient
from src.ai.anthropic_client import AnthropicClient
# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
app = FastAPI(
title="Izri AI Service",
description="AI-powered E2E testing and analysis service",
version="1.0.0"
)
# CORS middleware
app.add_middleware(
CORSMiddleware,
allow_origins=["*"], # Configure for production
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# Initialize AI clients
openai_client = None
anthropic_client = None
@app.on_event("startup")
async def startup_event():
global openai_client, anthropic_client
openai_key = os.getenv("OPENAI_API_KEY")
anthropic_key = os.getenv("ANTHROPIC_API_KEY")
if openai_key:
openai_client = OpenAIClient(api_key=openai_key)
logger.info("OpenAI client initialized")
if anthropic_key:
anthropic_client = AnthropicClient(api_key=anthropic_key)
logger.info("Anthropic client initialized")
class AnalysisRequest(BaseModel):
ai_ready_context: Dict[str, Any]
ai_provider: str = "openai"
ai_model: str = "gpt-4"
class TestGenerationRequest(BaseModel):
ai_ready_context: Dict[str, Any]
test_type: str = "e2e"
framework: str = "playwright"
ai_provider: str = "openai"
ai_model: str = "gpt-4"
@app.get("/")
async def root():
return {
"message": "Izri AI Service",
"version": "1.0.0",
"status": "healthy",
"ai_providers": {
"openai": openai_client is not None,
"anthropic": anthropic_client is not None
}
}
@app.get("/health")
async def health_check():
return {
"status": "healthy",
"timestamp": "2025-01-18T10:00:00.000Z",
"ai_providers": {
"openai": "available" if openai_client else "not_configured",
"anthropic": "available" if anthropic_client else "not_configured"
}
}
@app.post("/analyze")
async def analyze_repository(request: AnalysisRequest):
"""Analyze repository using AI"""
try:
client = get_ai_client(request.ai_provider)
if not client:
raise HTTPException(
status_code=400,
detail=f"AI provider '{request.ai_provider}' not available"
)
analysis = await client.analyze_repository(request.ai_ready_context)
return {
"status": "success",
"analysis": analysis,
"provider": request.ai_provider,
"model": request.ai_model,
"timestamp": "2025-01-18T10:00:00.000Z"
}
except Exception as e:
logger.error(f"Analysis failed: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.post("/generate-tests")
async def generate_tests(request: TestGenerationRequest):
"""Generate E2E tests using AI"""
try:
client = get_ai_client(request.ai_provider)
if not client:
raise HTTPException(
status_code=400,
detail=f"AI provider '{request.ai_provider}' not available"
)
tests = await client.generate_e2e_tests(
request.ai_ready_context,
request.test_type
)
return {
"status": "success",
"tests": tests,
"provider": request.ai_provider,
"model": request.ai_model,
"framework": request.framework,
"timestamp": "2025-01-18T10:00:00.000Z"
}
except Exception as e:
logger.error(f"Test generation failed: {e}")
raise HTTPException(status_code=500, detail=str(e))
def get_ai_client(provider: str):
"""Get AI client by provider"""
if provider == "openai":
return openai_client
elif provider == "anthropic":
return anthropic_client
else:
return None
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)
1.3 Basic E2E Test Execution Engine
Step 1.3.1: Create Test Executor Package
Commands:
# Create new package
mkdir -p packages/test-executor/src
# Create package.json
cat > packages/test-executor/package.json << 'EOF'
{
"name": "@izri/test-executor",
"version": "1.0.0",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"scripts": {
"build": "tsc",
"dev": "tsc --watch",
"test": "vitest"
},
"dependencies": {
"@izri/shared": "workspace:*"
},
"devDependencies": {
"typescript": "^5.0.0",
"vitest": "^1.0.0"
}
}
EOF
# Create TypeScript config
cat > packages/test-executor/tsconfig.json << 'EOF'
{
"extends": "../../tsconfig.json",
"compilerOptions": {
"outDir": "dist",
"rootDir": "src"
},
"include": ["src/**/*"],
"exclude": ["dist", "node_modules"]
}
EOF
Step 1.3.2: Implement Test Executor
Commands:
# Create executor implementation
touch packages/test-executor/src/executor.ts
touch packages/test-executor/src/types.ts
Implementation:
// File: packages/test-executor/src/types.ts
export interface TestExecutionOptions {
stagingUrl: string
framework: 'playwright' | 'cypress'
timeout?: number
headless?: boolean
browser?: 'chrome' | 'firefox' | 'webkit'
}
export interface TestExecutionResult {
success: boolean
results: TestResult[]
coverage?: number
logs: string
duration: number
error?: string
}
export interface TestResult {
name: string
file: string
status: 'passed' | 'failed' | 'skipped'
duration: number
error?: string
screenshots?: string[]
}
// File: packages/test-executor/src/executor.ts
import { spawn, ChildProcess } from 'child_process'
import { promises as fs } from 'fs'
import path from 'path'
import { TestExecutionOptions, TestExecutionResult, TestResult } from './types'
export class TestExecutor {
private workspaceDir: string
private dockerImage: string
constructor(workspaceDir = '/tmp/izri-execution') {
this.workspaceDir = workspaceDir
this.dockerImage = 'mcr.microsoft.com/playwright:v1.48.0-focal'
}
async executeTests(
generatedTests: any[],
options: TestExecutionOptions
): Promise<TestExecutionResult> {
const startTime = Date.now()
try {
// Create temporary workspace
await this.createWorkspace()
// Generate test files
await this.generateTestFiles(generatedTests)
// Execute tests in Docker
const result = await this.runTestsInDocker(options)
// Collect results
const duration = Date.now() - startTime
return {
success: result.exitCode === 0,
results: this.parseTestResults(result.output),
coverage: result.coverage,
logs: result.logs,
duration,
error: result.exitCode !== 0 ? result.error : undefined
}
} catch (error) {
const duration = Date.now() - startTime
return {
success: false,
results: [],
logs: '',
duration,
error: error instanceof Error ? error.message : 'Unknown error'
}
} finally {
await this.cleanup()
}
}
private async createWorkspace(): Promise<void> {
await fs.mkdir(this.workspaceDir, { recursive: true })
// Create package.json
const packageJson = {
name: 'izri-execution',
version: '1.0.0',
scripts: {
test: 'playwright test',
'test:headed': 'playwright test --headed'
},
devDependencies: {
'@playwright/test': '^1.48.0',
'typescript': '^5.0.0',
'@types/node': '^20.0.0'
}
}
await fs.writeFile(
path.join(this.workspaceDir, 'package.json'),
JSON.stringify(packageJson, null, 2)
)
// Create playwright config
const playwrightConfig = {
testDir: '.',
timeout: 30000,
expect: { timeout: 5000 },
fullyParallel: true,
forbidOnly: !!process.env.CI,
retries: process.env.CI ? 2 : 0,
workers: process.env.CI ? 1 : undefined,
reporter: 'json',
use: {
baseURL: 'STAGING_URL', // Will be replaced
trace: 'on-first-retry',
screenshot: 'only-on-failure',
video: 'retain-on-failure'
},
projects: [
{
name: 'chromium',
use: { ...require('playwright').devices['Desktop Chrome'] }
},
{
name: 'firefox',
use: { ...require('playwright').devices['Desktop Firefox'] }
},
{
name: 'webkit',
use: { ...require('playwright').devices['Desktop Safari'] }
}
],
webServer: {
command: 'echo "Server ready"',
port: 3000,
reuseExistingServer: !process.env.CI
}
}
await fs.writeFile(
path.join(this.workspaceDir, 'playwright.config.ts'),
`import { defineConfig } from '@playwright/test';
export default defineConfig(${JSON.stringify(playwrightConfig, null, 2)});`
)
}
private async generateTestFiles(generatedTests: any[]): Promise<void> {
for (const test of generatedTests) {
const fileName = test.fileName || 'generated-test.spec.ts'
const filePath = path.join(this.workspaceDir, fileName)
// Ensure directory exists
await fs.mkdir(path.dirname(filePath), { recursive: true })
// Write test file
await fs.writeFile(filePath, test.code || this.generateDefaultTest(test))
}
}
private generateDefaultTest(test: any): string {
return `
import { test, expect } from '@playwright/test';
test.describe('${test.description || 'Generated Test'}', () => {
test('should pass', async ({ page }) => {
// TODO: Implement test based on AI-generated requirements
await page.goto('STAGING_URL');
await expect(page).toHaveTitle(/.+/);
});
});
`.trim()
}
private async runTestsInDocker(options: TestExecutionOptions): Promise<any> {
return new Promise((resolve) => {
const dockerCommand = [
'docker', 'run', '--rm',
'-v', `${this.workspaceDir}:/app`,
'-e', `BASE_URL=${options.stagingUrl}`,
'-e', 'CI=true',
this.dockerImage,
'sh', '-c',
`
cd /app &&
npm install &&
npx playwright install &&
npx playwright test --reporter=json
`
]
const process = spawn(dockerCommand[0], dockerCommand.slice(1), {
stdio: ['pipe', 'pipe', 'pipe']
})
let output = ''
let errorOutput = ''
process.stdout?.on('data', (data) => {
output += data.toString()
})
process.stderr?.on('data', (data) => {
errorOutput += data.toString()
})
process.on('close', (code) => {
resolve({
exitCode: code,
output,
error: errorOutput,
logs: output + '\\n' + errorOutput
})
})
process.on('error', (error) => {
resolve({
exitCode: -1,
output: '',
error: error.message,
logs: error.message
})
})
})
}
private parseTestResults(output: string): TestResult[] {
try {
// Extract JSON report from output
const jsonMatch = output.match(/\\{.*"suites":.*\\}/s)
if (jsonMatch) {
const report = JSON.parse(jsonMatch[0])
return this.convertPlaywrightReportToResults(report)
}
// Fallback parsing
return this.parseTextResults(output)
} catch (error) {
console.error('Failed to parse test results:', error)
return []
}
}
private convertPlaywrightReportToResults(report: any): TestResult[] {
const results: TestResult[] = []
for (const suite of report.suites || []) {
for (const spec of suite.specs || []) {
for (const test of spec.tests || []) {
results.push({
name: test.title,
file: spec.file,
status: test.results[0]?.status || 'failed',
duration: test.results[0]?.duration || 0,
error: test.results[0]?.error?.message,
screenshots: test.results[0]?.attachments?.filter((a: any) => a.name === 'screenshot') || []
})
}
}
}
return results
}
private parseTextResults(output: string): TestResult[] {
const results: TestResult[] = []
const lines = output.split('\\n')
for (const line of lines) {
if (line.includes('✓') || line.includes('✗')) {
const status = line.includes('✓') ? 'passed' : 'failed'
const name = line.replace(/[✓✗]/, '').trim()
results.push({
name,
file: 'unknown',
status,
duration: 0
})
}
}
return results
}
private async cleanup(): Promise<void> {
try {
await fs.rm(this.workspaceDir, { recursive: true, force: true })
} catch (error) {
console.error('Cleanup failed:', error)
}
}
}
Step 1.3.3: Create Test Execution API
Commands:
# Create API router
touch apps/api/src/router/test-execution.ts
Implementation:
import { router, protectedProcedure } from '../trpc'
import { z } from 'zod'
import { TestExecutor } from '@izri/test-executor'
import { prisma } from '@izri/database'
export const testExecutionRouter = router({
executeTests: protectedProcedure
.input(z.object({
projectId: z.string(),
stagingUrl: z.string().url(),
generatedTests: z.array(z.any()),
framework: z.enum(['playwright', 'cypress']).default('playwright'),
options: z.object({
timeout: z.number().optional(),
headless: z.boolean().default(true),
browser: z.enum(['chrome', 'firefox', 'webkit']).optional()
}).optional()
}))
.mutation(async ({ input, ctx }) => {
const { projectId, stagingUrl, generatedTests, framework, options } = input
try {
// Create test run record
const testRun = await prisma.testRun.create({
data: {
projectId,
userId: ctx.user.id,
type: 'e2e',
status: 'RUNNING',
totalTests: generatedTests.length,
startedAt: new Date(),
results: {
stagingUrl,
framework,
generatedTests: generatedTests.length
}
}
})
// Execute tests
const executor = new TestExecutor()
const result = await executor.executeTests(generatedTests, {
stagingUrl,
framework,
...options
})
// Update test run
await prisma.testRun.update({
where: { id: testRun.id },
data: {
status: result.success ? 'COMPLETED' : 'FAILED',
completedAt: new Date(),
duration: result.duration,
passedTests: result.results.filter(r => r.status === 'passed').length,
failedTests: result.results.filter(r => r.status === 'failed').length,
skippedTests: result.results.filter(r => r.status === 'skipped').length,
coverage: result.coverage,
results: result.results,
logs: result.logs,
error: result.error
}
})
return {
runId: testRun.id,
status: result.success ? 'COMPLETED' : 'FAILED',
results: result.results,
duration: result.duration,
error: result.error
}
} catch (error) {
throw new Error(`Test execution failed: ${error}`)
}
}),
getTestRun: protectedProcedure
.input(z.object({ runId: z.string() }))
.query(async ({ input }) => {
const run = await prisma.testRun.findUnique({
where: { id: input.runId },
include: {
project: {
select: {
name: true,
repositoryUrl: true
}
}
}
})
if (!run) {
throw new Error('Test run not found')
}
return run
}),
listTestRuns: protectedProcedure
.input(z.object({
projectId: z.string(),
limit: z.number().default(20),
offset: z.number().default(0)
}))
.query(async ({ input }) => {
const { projectId, limit, offset } = input
const [runs, total] = await Promise.all([
prisma.testRun.findMany({
where: { projectId },
orderBy: { createdAt: 'desc' },
take: limit,
skip: offset,
include: {
project: {
select: {
name: true
}
}
}
}),
prisma.testRun.count({
where: { projectId }
})
])
return {
runs,
total,
hasMore: offset + runs.length < total
}
})
})
1.4 Integration & Testing
Step 1.4.1: Create Integration Test
Commands:
# Create integration test
touch apps/api/src/integration/e2e-workflow.test.ts
Implementation:
import { EnhancedRepositoryAnalyzer } from '@izri/repo-analyzer'
import { TestExecutor } from '@izri/test-executor'
import { describe, it, expect, beforeEach } from 'vitest'
describe('E2E Testing Workflow', () => {
let repoAnalyzer: EnhancedRepositoryAnalyzer
let testExecutor: TestExecutor
beforeEach(() => {
repoAnalyzer = new EnhancedRepositoryAnalyzer()
testExecutor = new TestExecutor()
})
it('should complete full E2E testing workflow', async () => {
// Step 1: Analyze repository
const analysis = await repoAnalyzer.analyzeForAI(
'https://github.com/example/react-app',
{ branch: 'main' }
)
expect(analysis).toBeDefined()
expect(analysis.staging).toBeDefined()
expect(analysis.codebase).toBeDefined()
// Step 2: Generate tests (mock AI service for testing)
const mockGeneratedTests = [
{
fileName: 'auth/login.spec.ts',
description: 'User login flow',
code: `
import { test, expect } from '@playwright/test';
test('should login successfully', async ({ page }) => {
await page.goto('${analysis.staging.recommendedUrl || 'https://staging.example.com'}');
await page.fill('[data-testid=email]', 'test@example.com');
await page.fill('[data-testid=password]', 'password123');
await page.click('[data-testid=login-button]');
await expect(page.locator('[data-testid=dashboard]')).toBeVisible();
});
`
}
]
// Step 3: Execute tests
const result = await testExecutor.executeTests(mockGeneratedTests, {
stagingUrl: analysis.staging.recommendedUrl || 'https://staging.example.com',
framework: 'playwright'
})
expect(result).toBeDefined()
expect(result.results).toHaveLength(1)
})
})
🧪 Testing Strategy
Unit Testing
# Test individual components
pnpm --filter @izri/repo-analyzer test
pnpm --filter @izri/test-executor test
pnpm --filter @izri/api test
Integration Testing
# Test end-to-end workflow
pnpm --filter @izri/api test:integration
Manual Testing
# Start services
docker-compose up -d
# Test repository analysis
curl -X POST http://localhost:3000/trpc/repo.analyzeRepository \
-H "Content-Type: application/json" \
-d '{"repoUrl": "https://github.com/example/repo"}'
# Test AI service
curl -X POST http://localhost:8000/generate-tests \
-H "Content-Type: application/json" \
-d '{"ai_ready_context": {...}, "test_type": "e2e"}'
🚀 Deployment Instructions
Development Setup
# Install dependencies
pnpm install
# Build packages
pnpm build
# Start AI service
cd packages/ai-service
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python main.py
# Run tests
pnpm test
Production Deployment
# Build Docker images
docker build -t izri-ai-service ./packages/ai-service
docker build -t izri-api ./apps/api
# Deploy with Docker Compose
docker-compose -f docker-compose.yml up -d
🔍 Troubleshooting
Common Issues
- AI Service Connection: Check API keys and network connectivity
- Docker Test Execution: Verify Docker daemon and image availability
- Repository Cloning: Ensure GitHub tokens have proper permissions
- Memory Issues: Adjust
maxFilesandmaxFileSizeBytesin analysis options
Debug Commands
# Check AI service health
curl http://localhost:8000/health
# Test repository analysis
pnpm --filter @izri/repo-analyzer test
# Verify Docker execution
docker run --rm mcr.microsoft.com/playwright:v1.48.0-focal npx playwright --version
Next Steps: Proceed to Phase 2: CI/CD & Diff Analysis Document Status: ✅ Ready for Implementation Last Updated: 2025-01-18 Implementation Priority: HIGH