E2E Testing MVP Implementation Plan
Detailed step-by-step implementation guide for AI agents and developers
🎯 Prerequisites
Before starting implementation, ensure you have:
- Node.js 18+ and pnpm installed
- Docker and Docker Compose installed
- Python 3.11+ for AI service
- Access to OpenAI/Anthropic API keys (for testing)
- GitHub token for private repository testing
🏗️ Phase 1: Core MVP Foundation (Weeks 1-3)
1.1 Repository Analysis Extensions
Step 1.1.1: Create Staging Environment Plugin
File: packages/repo-analyzer/src/analysis/plugins/staging-environment.ts
import { AnalysisPlugin, RepoContext, AnalysisResult } from '../types/intake'
export class StagingEnvironmentPlugin implements AnalysisPlugin {
id = 'staging-environment'
name = 'Staging Environment Detection'
description = 'Detects and configures staging environment URLs for E2E testing'
supportedExtensions = ['.json', '.yaml', '.yml', '.env', '.md']
async analyze(context: RepoContext): Promise<AnalysisResult> {
const stagingUrls = await this.detectStagingUrls(context)
const environments = await this.detectEnvironments(context)
const configurations = await this.detectConfigurations(context)
return {
pluginId: this.id,
metrics: {
stagingUrlsFound: stagingUrls.length,
environmentsDetected: environments.length,
hasStagingConfig: configurations.length > 0
},
issues: this.validateStagingConfig(stagingUrls, configurations),
metadata: {
stagingUrls,
environments,
configurations,
recommendedUrl: this.recommendStagingUrl(stagingUrls)
},
analyzedAt: new Date().toISOString()
}
}
private async detectStagingUrls(context: RepoContext): Promise<Array<{
url: string
source: string
confidence: number
}>> {
const urls: Array<{ url: string; source: string; confidence: number }> = []
// Scan common configuration files
const configFiles = [
'package.json',
'docker-compose.yml',
'docker-compose.yaml',
'.env.example',
'.env.staging',
'README.md',
'staging.md'
]
for (const file of configFiles) {
const fileContent = await this.readFileContent(context, file)
if (fileContent) {
const foundUrls = this.extractUrls(fileContent)
urls.push(...foundUrls.map(url => ({
url,
source: file,
confidence: this.calculateUrlConfidence(url, file)
})))
}
}
return urls.sort((a, b) => b.confidence - a.confidence)
}
private async detectEnvironments(context: RepoContext): Promise<string[]> {
// Detect common environment patterns
const environments = ['staging', 'dev', 'development', 'test', 'preview']
// Check for environment-specific configurations
const detected: string[] = []
for (const env of environments) {
const hasEnvConfig = context.fileIndex.files.some(file =>
file.path.toLowerCase().includes(env) ||
file.name.toLowerCase().includes(env)
)
if (hasEnvConfig) {
detected.push(env)
}
}
return detected
}
private async detectConfigurations(context: RepoContext): Promise<Array<{
type: string
file: string
content: any
}>> {
const configurations = []
// Check for various configuration types
const configPatterns = {
'docker-compose': ['docker-compose.yml', 'docker-compose.yaml'],
'environment': ['.env.staging', '.env.test'],
'deployment': ['vercel.json', 'netlify.toml', 'amplify.yml'],
'ci-cd': ['.github/workflows/staging.yml', '.gitlab-ci.yml']
}
for (const [type, files] of Object.entries(configPatterns)) {
for (const file of files) {
const content = await this.readFileContent(context, file)
if (content) {
try {
const parsed = file.endsWith('.json') ? JSON.parse(content) :
file.endsWith('.yml') || file.endsWith('.yaml') ?
this.parseYaml(content) : content
configurations.push({ type, file, content: parsed })
} catch (error) {
// Skip invalid configurations
}
}
}
}
return configurations
}
private extractUrls(content: string): string[] {
const urlPatterns = [
/https?:\/\/[a-zA-Z0-9-]+\.staging\.[a-zA-Z0-9.-]+/g,
/https?:\/\/staging\.[a-zA-Z0-9.-]+/g,
/https?:\/\/[a-zA-Z0-9-]+-staging\.[a-zA-Z0-9.-]+/g,
/https?:\/\/[a-zA-Z0-9.-]+\/staging/g
]
const urls = new Set<string>()
for (const pattern of urlPatterns) {
const matches = content.match(pattern)
if (matches) {
matches.forEach(url => urls.add(url))
}
}
return Array.from(urls)
}
private calculateUrlConfidence(url: string, source: string): number {
let confidence = 0.5
// Higher confidence for explicit staging URLs
if (url.includes('staging')) confidence += 0.3
// Higher confidence for configuration files
if (['package.json', 'docker-compose.yml', '.env.staging'].includes(source)) {
confidence += 0.2
}
return Math.min(confidence, 1.0)
}
private recommendStagingUrl(urls: Array<{ url: string; confidence: number }>): string | null {
if (urls.length === 0) return null
return urls.reduce((best, current) =>
current.confidence > best.confidence ? current : best
).url
}
private validateStagingConfig(urls: any[], configurations: any[]): any[] {
const issues = []
if (urls.length === 0) {
issues.push({
severity: 'warning',
category: 'staging',
message: 'No staging URLs detected. Manual configuration may be required.'
})
}
const hasHttps = urls.every(url => url.url?.startsWith?.('https://'))
if (!hasHttps) {
issues.push({
severity: 'warning',
category: 'security',
message: 'HTTP URLs detected. Consider using HTTPS for staging environments.'
})
}
return issues
}
private async readFileContent(context: RepoContext, filePath: string): Promise<string | null> {
try {
const fullPath = `${context.repoDir}/${filePath}`
// Implementation would read file from repository
return null // Placeholder
} catch {
return null
}
}
private parseYaml(content: string): any {
// Simple YAML parser - in real implementation use js-yaml
return {}
}
}
Testing: Create test file packages/repo-analyzer/src/analysis/__tests__/staging-environment-plugin.test.ts
import { StagingEnvironmentPlugin } from '../staging-environment-plugin'
import { createMockRepoContext } from '../../../test/utils/repo-context'
describe('StagingEnvironmentPlugin', () => {
let plugin: StagingEnvironmentPlugin
let mockContext: any
beforeEach(() => {
plugin = new StagingEnvironmentPlugin()
mockContext = createMockRepoContext()
})
it('should detect staging URLs from package.json', async () => {
// Test implementation
})
it('should validate staging configuration', async () => {
// Test implementation
})
})
Step 1.1.2: Extend AIReadyContext Interface
File: packages/repo-analyzer/src/types/intake.ts
Add to existing AIReadyContext interface:
export interface AIReadyContext {
// ... existing fields ...
// New staging environment fields
staging: {
urls: Array<{
url: string
source: string
confidence: number
}>
environments: string[]
configurations: Array<{
type: string
file: string
content: any
}>
recommendedUrl?: string
hasStagingConfig: boolean
}
}
Step 1.1.3: Update EnhancedAnalyzer
File: packages/repo-analyzer/src/enhanced-analyzer.ts
Add staging plugin registration:
import { StagingEnvironmentPlugin } from './analysis/staging-environment-plugin'
export class EnhancedRepositoryAnalyzer {
constructor(additionalPlugins: AnalysisPlugin[] = []) {
this.intake = new RepositoryIntake()
// Add staging plugin to default plugins
const defaultPlugins = [
new StagingEnvironmentPlugin(), // Add this
// ... existing plugins
]
this.traditionalPipeline = new TraditionalAnalysisPipeline([...defaultPlugins, ...additionalPlugins])
}
// Update prepareAIContext method to include staging information
private prepareAIContext(analysis: ComprehensiveAnalysis): AIReadyContext {
// ... existing code ...
const stagingResult = results.find((r) => r.pluginId === 'staging-environment')
return {
// ... existing fields ...
staging: {
urls: stagingResult?.metadata.stagingUrls || [],
environments: stagingResult?.metadata.environments || [],
configurations: stagingResult?.metadata.configurations || [],
recommendedUrl: stagingResult?.metadata.recommendedUrl,
hasStagingConfig: stagingResult?.metrics.hasStagingConfig || false
}
}
}
}
1.2 AI Service Real Integration
Step 1.2.1: Install AI SDK Dependencies
File: packages/ai-service/requirements.txt
fastapi==0.115.6
uvicorn[standard]==0.34.0
pydantic==2.10.3
openai==1.58.1
anthropic==0.40.0
python-dotenv==1.0.1
httpx==0.28.1
langchain==0.3.0
langchain-openai==0.2.0
Step 1.2.2: Implement Real AI Integration
File: packages/ai-service/src/ai/openai_client.py
from typing import Dict, Any, List, Optional
import openai
from openai import OpenAI
import json
import logging
logger = logging.getLogger(__name__)
class OpenAIClient:
def __init__(self, api_key: str, model: str = "gpt-4"):
self.client = OpenAI(api_key=api_key)
self.model = model
async def analyze_repository(self, ai_ready_context: Dict[str, Any]) -> Dict[str, Any]:
"""Analyze repository using OpenAI GPT-4"""
prompt = self._build_analysis_prompt(ai_ready_context)
try:
response = await self.client.chat.completions.acreate(
model=self.model,
messages=[
{"role": "system", "content": "You are an expert software testing consultant."},
{"role": "user", "content": prompt}
],
max_tokens=2000,
temperature=0.3
)
analysis_text = response.choices[0].message.content
return self._parse_analysis_response(analysis_text, ai_ready_context)
except Exception as e:
logger.error(f"OpenAI analysis failed: {e}")
raise
async def generate_e2e_tests(self, ai_ready_context: Dict[str, Any], test_type: str = "e2e") -> Dict[str, Any]:
"""Generate E2E tests based on repository analysis"""
prompt = self._build_test_generation_prompt(ai_ready_context, test_type)
try:
response = await self.client.chat.completions.acreate(
model=self.model,
messages=[
{"role": "system", "content": "You are an expert test automation engineer specializing in E2E testing with Playwright."},
{"role": "user", "content": prompt}
],
max_tokens=3000,
temperature=0.2
)
tests_text = response.choices[0].message.content
return self._parse_test_generation_response(tests_text, ai_ready_context)
except Exception as e:
logger.error(f"Test generation failed: {e}")
raise
def _build_analysis_prompt(self, context: Dict[str, Any]) -> str:
"""Build comprehensive analysis prompt"""
repo_info = context.get('repository', {})
project_info = context.get('project', {})
codebase_info = context.get('codebase', {})
testing_info = context.get('testing', {})
staging_info = context.get('staging', {})
prompt = f"""
Analyze this repository for E2E testing strategy:
REPOSITORY INFO:
- URL: {repo_info.get('url', 'Unknown')}
- Branch: {repo_info.get('branch', 'main')}
- Commit: {repo_info.get('commit', 'latest')}
PROJECT DETAILS:
- Type: {project_info.get('type', 'Unknown')}
- Frameworks: {', '.join(project_info.get('frameworks', []))}
- Testing Frameworks: {', '.join(project_info.get('testingFrameworks', []))}
- Build Tools: {', '.join(project_info.get('buildTools', []))}
CODEBASE STRUCTURE:
- Total Files: {codebase_info.get('totalFiles', 0)}
- Languages: {', '.join(codebase_info.get('languages', {}).keys())}
- Entry Points: {', '.join(codebase_info.get('structure', {}).get('entryPoints', []))}
- API Endpoints: {len(codebase_info.get('structure', {}).get('apiEndpoints', []))} found
TESTING STATUS:
- Has Tests: {testing_info.get('hasTests', False)}
- Test Files: {len(testing_info.get('testFiles', []))}
- Test Frameworks: {', '.join(testing_info.get('testFrameworks', []))}
- Test to Source Ratio: {testing_info.get('testToSourceRatio', 0):.2f}
STAGING ENVIRONMENT:
- Has Staging Config: {staging_info.get('hasStagingConfig', False)}
- Recommended URL: {staging_info.get('recommendedUrl', 'None detected')}
- Environments: {', '.join(staging_info.get('environments', []))}
Based on this analysis, provide:
1. E2E testing strategy recommendations
2. Critical user flows to test
3. Technical implementation approach
4. Potential challenges and solutions
5. Test framework recommendations (Playwright vs Cypress)
Respond in JSON format with keys: strategy, criticalFlows, implementation, challenges, recommendations.
"""
return prompt
def _build_test_generation_prompt(self, context: Dict[str, Any], test_type: str) -> str:
"""Build test generation prompt"""
project_info = context.get('project', {})
codebase_info = context.get('codebase', {})
staging_info = context.get('staging', {})
base_url = staging_info.get('recommendedUrl', 'https://staging.example.com')
frameworks = project_info.get('frameworks', [])
prompt = f"""
Generate comprehensive E2E tests for this application:
APPLICATION DETAILS:
- Frameworks: {', '.join(frameworks)}
- Base URL: {base_url}
- Test Type: {test_type}
Generate Playwright tests in TypeScript that cover:
1. User authentication flows
2. Core user journeys
3. Form submissions and validation
4. Navigation and routing
5. Error handling scenarios
For each test, provide:
- Test file name
- Complete Playwright test code
- Test description and what it validates
- Expected test data setup
Focus on:
- Cross-browser compatibility (Chrome, Firefox, Safari)
- Mobile responsiveness
- Accessibility testing
- Performance considerations
Respond in JSON format with structure:
{{
"tests": [
{{
"fileName": "auth/login.spec.ts",
"description": "User login flow validation",
"code": "complete Playwright test code here",
"testData": {{"email": "test@example.com", "password": "test123"}},
"validates": ["login functionality", "error handling", "redirect after login"]
}}
],
"setupRequired": ["test users", "test data", "environment variables"],
"dependencies": ["@playwright/test", "typescript"],
"estimatedExecutionTime": "2-3 minutes"
}}
"""
return prompt
def _parse_analysis_response(self, response_text: str, context: Dict[str, Any]) -> Dict[str, Any]:
"""Parse OpenAI analysis response"""
try:
# Try to extract JSON from response
import re
json_match = re.search(r'\\{.*\\}', response_text, re.DOTALL)
if json_match:
return json.loads(json_match.group())
else:
# Fallback to structured response
return {
"strategy": response_text,
"criticalFlows": ["Manual review required"],
"implementation": "Based on analysis",
"challenges": ["Response parsing required"],
"recommendations": ["Review AI response"]
}
except Exception as e:
logger.error(f"Failed to parse analysis response: {e}")
return {"error": "Failed to parse AI response", "raw_response": response_text}
def _parse_test_generation_response(self, response_text: str, context: Dict[str, Any]) -> Dict[str, Any]:
"""Parse test generation response"""
try:
import re
json_match = re.search(r'\\{.*\\}', response_text, re.DOTALL)
if json_match:
return json.loads(json_match.group())
else:
return {
"tests": [],
"setupRequired": [],
"dependencies": [],
"estimatedExecutionTime": "Unknown",
"error": "Failed to parse test generation response"
}
except Exception as e:
logger.error(f"Failed to parse test generation response: {e}")
return {"error": "Failed to parse test response", "raw_response": response_text}
Step 1.2.3: Update Main FastAPI Application
File: packages/ai-service/main.py
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from typing import Dict, Any, List, Optional
import os
import logging
from src.ai.openai_client import OpenAIClient
from src.ai.anthropic_client import AnthropicClient
# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
app = FastAPI(
title="Izri AI Service",
description="AI-powered E2E testing and analysis service",
version="1.0.0"
)
# CORS middleware
app.add_middleware(
CORSMiddleware,
allow_origins=["*"], # Configure for production
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# Initialize AI clients
openai_client = None
anthropic_client = None
@app.on_event("startup")
async def startup_event():
global openai_client, anthropic_client
openai_key = os.getenv("OPENAI_API_KEY")
anthropic_key = os.getenv("ANTHROPIC_API_KEY")
if openai_key:
openai_client = OpenAIClient(api_key=openai_key)
logger.info("OpenAI client initialized")
if anthropic_key:
anthropic_client = AnthropicClient(api_key=anthropic_key)
logger.info("Anthropic client initialized")
class AnalysisRequest(BaseModel):
ai_ready_context: Dict[str, Any]
ai_provider: str = "openai"
ai_model: str = "gpt-4"
class TestGenerationRequest(BaseModel):
ai_ready_context: Dict[str, Any]
test_type: str = "e2e"
framework: str = "playwright"
ai_provider: str = "openai"
ai_model: str = "gpt-4"
@app.get("/")
async def root():
return {
"message": "Izri AI Service",
"version": "1.0.0",
"status": "healthy",
"ai_providers": {
"openai": openai_client is not None,
"anthropic": anthropic_client is not None
}
}
@app.get("/health")
async def health_check():
return {
"status": "healthy",
"timestamp": "2025-01-18T10:00:00.000Z",
"ai_providers": {
"openai": "available" if openai_client else "not_configured",
"anthropic": "available" if anthropic_client else "not_configured"
}
}
@app.post("/analyze")
async def analyze_repository(request: AnalysisRequest):
"""Analyze repository using AI"""
try:
client = get_ai_client(request.ai_provider)
if not client:
raise HTTPException(
status_code=400,
detail=f"AI provider '{request.ai_provider}' not available"
)
analysis = await client.analyze_repository(request.ai_ready_context)
return {
"status": "success",
"analysis": analysis,
"provider": request.ai_provider,
"model": request.ai_model,
"timestamp": "2025-01-18T10:00:00.000Z"
}
except Exception as e:
logger.error(f"Analysis failed: {e}")
raise HTTPException(status_code=500, detail=str(e))
@app.post("/generate-tests")
async def generate_tests(request: TestGenerationRequest):
"""Generate E2E tests using AI"""
try:
client = get_ai_client(request.ai_provider)
if not client:
raise HTTPException(
status_code=400,
detail=f"AI provider '{request.ai_provider}' not available"
)
tests = await client.generate_e2e_tests(
request.ai_ready_context,
request.test_type
)
return {
"status": "success",
"tests": tests,
"provider": request.ai_provider,
"model": request.ai_model,
"framework": request.framework,
"timestamp": "2025-01-18T10:00:00.000Z"
}
except Exception as e:
logger.error(f"Test generation failed: {e}")
raise HTTPException(status_code=500, detail=str(e))
def get_ai_client(provider: str):
"""Get AI client by provider"""
if provider == "openai":
return openai_client
elif provider == "anthropic":
return anthropic_client
else:
return None
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)
1.3 Basic E2E Test Execution Engine
Step 1.3.1: Create Test Executor Service
File: packages/test-executor/src/executor.ts
import { spawn, ChildProcess } from 'child_process'
import { promises as fs } from 'fs'
import path from 'path'
import { TestRun, TestResult } from '../types'
export interface TestExecutionOptions {
stagingUrl: string
framework: 'playwright' | 'cypress'
timeout?: number
headless?: boolean
browser?: 'chrome' | 'firefox' | 'webkit'
}
export interface TestExecutionResult {
success: boolean
results: TestResult[]
coverage?: number
logs: string
duration: number
error?: string
}
export class TestExecutor {
private workspaceDir: string
private dockerImage: string
constructor(workspaceDir = '/tmp/izri-execution') {
this.workspaceDir = workspaceDir
this.dockerImage = 'mcr.microsoft.com/playwright:v1.48.0-focal'
}
async executeTests(
generatedTests: any[],
options: TestExecutionOptions
): Promise<TestExecutionResult> {
const startTime = Date.now()
try {
// Create temporary workspace
await this.createWorkspace()
// Generate test files
await this.generateTestFiles(generatedTests)
// Execute tests in Docker
const result = await this.runTestsInDocker(options)
// Collect results
const duration = Date.now() - startTime
return {
success: result.exitCode === 0,
results: this.parseTestResults(result.output),
coverage: result.coverage,
logs: result.logs,
duration,
error: result.exitCode !== 0 ? result.error : undefined
}
} catch (error) {
const duration = Date.now() - startTime
return {
success: false,
results: [],
logs: '',
duration,
error: error instanceof Error ? error.message : 'Unknown error'
}
} finally {
await this.cleanup()
}
}
private async createWorkspace(): Promise<void> {
await fs.mkdir(this.workspaceDir, { recursive: true })
// Create package.json
const packageJson = {
name: 'izri-execution',
version: '1.0.0',
scripts: {
test: 'playwright test',
'test:headed': 'playwright test --headed'
},
devDependencies: {
'@playwright/test': '^1.48.0',
'typescript': '^5.0.0',
'@types/node': '^20.0.0'
}
}
await fs.writeFile(
path.join(this.workspaceDir, 'package.json'),
JSON.stringify(packageJson, null, 2)
)
// Create playwright config
const playwrightConfig = {
testDir: '.',
timeout: 30000,
expect: { timeout: 5000 },
fullyParallel: true,
forbidOnly: !!process.env.CI,
retries: process.env.CI ? 2 : 0,
workers: process.env.CI ? 1 : undefined,
reporter: 'json',
use: {
baseURL: 'STAGING_URL', // Will be replaced
trace: 'on-first-retry',
screenshot: 'only-on-failure',
video: 'retain-on-failure'
},
projects: [
{
name: 'chromium',
use: { ...require('playwright').devices['Desktop Chrome'] }
},
{
name: 'firefox',
use: { ...require('playwright').devices['Desktop Firefox'] }
},
{
name: 'webkit',
use: { ...require('playwright').devices['Desktop Safari'] }
}
],
webServer: {
command: 'echo "Server ready"',
port: 3000,
reuseExistingServer: !process.env.CI
}
}
await fs.writeFile(
path.join(this.workspaceDir, 'playwright.config.ts'),
`import { defineConfig } from '@playwright/test';
export default defineConfig(${JSON.stringify(playwrightConfig, null, 2)});`
)
}
private async generateTestFiles(generatedTests: any[]): Promise<void> {
for (const test of generatedTests) {
const fileName = test.fileName || 'generated-test.spec.ts'
const filePath = path.join(this.workspaceDir, fileName)
// Ensure directory exists
await fs.mkdir(path.dirname(filePath), { recursive: true })
// Write test file
await fs.writeFile(filePath, test.code || this.generateDefaultTest(test))
}
}
private generateDefaultTest(test: any): string {
return `
import { test, expect } from '@playwright/test';
test.describe('${test.description || 'Generated Test'}', () => {
test('should pass', async ({ page }) => {
// TODO: Implement test based on AI-generated requirements
await page.goto('STAGING_URL');
await expect(page).toHaveTitle(/.+/);
});
});
`.trim()
}
private async runTestsInDocker(options: TestExecutionOptions): Promise<any> {
return new Promise((resolve) => {
const dockerCommand = [
'docker', 'run', '--rm',
'-v', `${this.workspaceDir}:/app`,
'-e', `BASE_URL=${options.stagingUrl}`,
'-e', 'CI=true',
this.dockerImage,
'sh', '-c',
`
cd /app &&
npm install &&
npx playwright install &&
npx playwright test --reporter=json
`
]
const process = spawn(dockerCommand[0], dockerCommand.slice(1), {
stdio: ['pipe', 'pipe', 'pipe']
})
let output = ''
let errorOutput = ''
process.stdout?.on('data', (data) => {
output += data.toString()
})
process.stderr?.on('data', (data) => {
errorOutput += data.toString()
})
process.on('close', (code) => {
resolve({
exitCode: code,
output,
error: errorOutput,
logs: output + '\\n' + errorOutput
})
})
process.on('error', (error) => {
resolve({
exitCode: -1,
output: '',
error: error.message,
logs: error.message
})
})
})
}
private parseTestResults(output: string): TestResult[] {
try {
// Extract JSON report from output
const jsonMatch = output.match(/\\{.*"suites":.*\\}/s)
if (jsonMatch) {
const report = JSON.parse(jsonMatch[0])
return this.convertPlaywrightReportToResults(report)
}
// Fallback parsing
return this.parseTextResults(output)
} catch (error) {
console.error('Failed to parse test results:', error)
return []
}
}
private convertPlaywrightReportToResults(report: any): TestResult[] {
const results: TestResult[] = []
for (const suite of report.suites || []) {
for (const spec of suite.specs || []) {
for (const test of spec.tests || []) {
results.push({
name: test.title,
file: spec.file,
status: test.results[0]?.status || 'failed',
duration: test.results[0]?.duration || 0,
error: test.results[0]?.error?.message,
screenshots: test.results[0]?.attachments?.filter((a: any) => a.name === 'screenshot') || []
})
}
}
}
return results
}
private parseTextResults(output: string): TestResult[] {
const results: TestResult[] = []
const lines = output.split('\\n')
for (const line of lines) {
if (line.includes('✓') || line.includes('✗')) {
const status = line.includes('✓') ? 'passed' : 'failed'
const name = line.replace(/[✓✗]/, '').trim()
results.push({
name,
file: 'unknown',
status,
duration: 0
})
}
}
return results
}
private async cleanup(): Promise<void> {
try {
await fs.rm(this.workspaceDir, { recursive: true, force: true })
} catch (error) {
console.error('Cleanup failed:', error)
}
}
}
Step 1.3.2: Create Test Execution API
File: packages/api/src/router/test-execution.ts
import { router, protectedProcedure } from '../trpc'
import { z } from 'zod'
import { TestExecutor } from '@izri/test-executor'
import { prisma } from '@izri/database'
export const testExecutionRouter = router({
executeTests: protectedProcedure
.input(z.object({
projectId: z.string(),
stagingUrl: z.string().url(),
generatedTests: z.array(z.any()),
framework: z.enum(['playwright', 'cypress']).default('playwright'),
options: z.object({
timeout: z.number().optional(),
headless: z.boolean().default(true),
browser: z.enum(['chrome', 'firefox', 'webkit']).optional()
}).optional()
}))
.mutation(async ({ input, ctx }) => {
const { projectId, stagingUrl, generatedTests, framework, options } = input
try {
// Create test run record
const testRun = await prisma.testRun.create({
data: {
projectId,
userId: ctx.user.id,
type: 'e2e',
status: 'RUNNING',
totalTests: generatedTests.length,
startedAt: new Date(),
results: {
stagingUrl,
framework,
generatedTests: generatedTests.length
}
}
})
// Execute tests
const executor = new TestExecutor()
const result = await executor.executeTests(generatedTests, {
stagingUrl,
framework,
...options
})
// Update test run
await prisma.testRun.update({
where: { id: testRun.id },
data: {
status: result.success ? 'COMPLETED' : 'FAILED',
completedAt: new Date(),
duration: result.duration,
passedTests: result.results.filter(r => r.status === 'passed').length,
failedTests: result.results.filter(r => r.status === 'failed').length,
skippedTests: result.results.filter(r => r.status === 'skipped').length,
coverage: result.coverage,
results: result.results,
logs: result.logs,
error: result.error
}
})
return {
runId: testRun.id,
status: result.success ? 'COMPLETED' : 'FAILED',
results: result.results,
duration: result.duration,
error: result.error
}
} catch (error) {
throw new Error(`Test execution failed: ${error}`)
}
}),
getTestRun: protectedProcedure
.input(z.object({ runId: z.string() }))
.query(async ({ input }) => {
const run = await prisma.testRun.findUnique({
where: { id: input.runId },
include: {
project: {
select: {
name: true,
repositoryUrl: true
}
}
}
})
if (!run) {
throw new Error('Test run not found')
}
return run
}),
listTestRuns: protectedProcedure
.input(z.object({
projectId: z.string(),
limit: z.number().default(20),
offset: z.number().default(0)
}))
.query(async ({ input }) => {
const { projectId, limit, offset } = input
const [runs, total] = await Promise.all([
prisma.testRun.findMany({
where: { projectId },
orderBy: { createdAt: 'desc' },
take: limit,
skip: offset,
include: {
project: {
select: {
name: true
}
}
}
}),
prisma.testRun.count({
where: { projectId }
})
])
return {
runs,
total,
hasMore: offset + runs.length < total
}
})
})
1.4 Integration & Testing
Step 1.4.1: Create End-to-End Integration Test
File: apps/api/src/integration/e2e-workflow.test.ts
import { EnhancedRepositoryAnalyzer } from '@izri/repo-analyzer'
import { TestExecutor } from '@izri/test-executor'
import { describe, it, expect, beforeEach } from 'vitest'
describe('E2E Testing Workflow', () => {
let repoAnalyzer: EnhancedRepositoryAnalyzer
let testExecutor: TestExecutor
beforeEach(() => {
repoAnalyzer = new EnhancedRepositoryAnalyzer()
testExecutor = new TestExecutor()
})
it('should complete full E2E testing workflow', async () => {
// Step 1: Analyze repository
const analysis = await repoAnalyzer.analyzeForAI(
'https://github.com/example/react-app',
{ branch: 'main' }
)
expect(analysis).toBeDefined()
expect(analysis.staging).toBeDefined()
expect(analysis.codebase).toBeDefined()
// Step 2: Generate tests (mock AI service for testing)
const mockGeneratedTests = [
{
fileName: 'auth/login.spec.ts',
description: 'User login flow',
code: `
import { test, expect } from '@playwright/test';
test('should login successfully', async ({ page }) => {
await page.goto('${analysis.staging.recommendedUrl || 'https://staging.example.com'}');
await page.fill('[data-testid=email]', 'test@example.com');
await page.fill('[data-testid=password]', 'password123');
await page.click('[data-testid=login-button]');
await expect(page.locator('[data-testid=dashboard]')).toBeVisible();
});
`
}
]
// Step 3: Execute tests
const result = await testExecutor.executeTests(mockGeneratedTests, {
stagingUrl: analysis.staging.recommendedUrl || 'https://staging.example.com',
framework: 'playwright'
})
expect(result).toBeDefined()
expect(result.results).toHaveLength(1)
})
})
Step 1.4.2: Update Project Package Dependencies
File: packages/repo-analyzer/package.json
{
"dependencies": {
"@izri/shared": "workspace:*",
"simple-git": "^3.20.0",
"glob": "^10.3.0",
"fs-extra": "^11.1.0",
"js-yaml": "^4.1.0"
}
}
File: packages/test-executor/package.json (new package)
{
"name": "@izri/test-executor",
"version": "1.0.0",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"scripts": {
"build": "tsc",
"dev": "tsc --watch",
"test": "vitest"
},
"dependencies": {
"@izri/shared": "workspace:*"
},
"devDependencies": {
"typescript": "^5.0.0",
"vitest": "^1.0.0"
}
}
🧪 Testing Strategy
Unit Testing
- Each plugin unit tested with mock repository contexts
- AI client integration tested with mocked responses
- Test executor components tested in isolation
Integration Testing
- End-to-end workflow tested with sample repositories
- AI service integration tested with real API keys (in CI)
- Docker execution tested with test containers
Manual Testing
- Verify test generation with various repository types
- Test staging URL detection with real projects
- Validate test execution in different browsers
📊 Success Criteria
Phase 1 Completion Checklist
- Repository analysis detects staging environments
- AI service generates relevant E2E tests
- Test execution runs successfully against staging URLs
- Integration tests pass consistently
- Documentation is complete and accurate
- Performance targets met (< 5 min analysis, < 10 min execution)
🚀 Deployment Instructions
Development Setup
# Install dependencies
pnpm install
# Build packages
pnpm build
# Start AI service
cd packages/ai-service
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python main.py
# Run tests
pnpm test
Production Deployment
# Build Docker images
docker build -t izri-ai-service ./packages/ai-service
docker build -t izri-api ./apps/api
# Deploy with Docker Compose
docker-compose -f docker-compose.yml up -d
🔍 Troubleshooting
Common Issues
- AI Service Connection: Check API keys and network connectivity
- Docker Test Execution: Verify Docker daemon and image availability
- Repository Cloning: Ensure GitHub tokens have proper permissions
- Memory Issues: Adjust
maxFilesandmaxFileSizeBytesin analysis options
Debug Commands
# Check AI service health
curl http://localhost:8000/health
# Test repository analysis
pnpm --filter @izri/repo-analyzer test
# Verify Docker execution
docker run --rm mcr.microsoft.com/playwright:v1.48.0-focal npx playwright --version
Next Steps: Proceed to Phase 2: CI/CD & Diff Analysis Document Status: ✅ Ready for Implementation Last Updated: 2025-01-18 Implementation Priority: HIGH