# Production Deployment

> Production-ready deployment strategies and best practices

> **Note:** This is a reference doc covering generic deployment strategies (blue-green, canary, Kubernetes, etc.). For Izri's actual production setup — Railway with tag-triggered deploys via GitHub Actions — see [`railway.md`](./railway.md). The sections below are kept for context if the deployment target ever changes.

## 📋 Overview

This guide covers deploying Izri to production environments, including pre-deployment checklists, deployment strategies, rollback procedures, and monitoring.

## ✅ Pre-Deployment Checklist

### Code & Build

- [ ] All tests passing
- [ ] Code reviewed and approved
- [ ] Dependency vulnerabilities checked (`pnpm audit`)
- [ ] Production build tested locally
- [ ] Environment variables documented
- [ ] Database migrations ready
- [ ] No `.only` or `skip` in tests
- [ ] Console logs removed or gated by log level

### Infrastructure

- [ ] Database backed up
- [ ] Secrets stored securely
- [ ] SSL/TLS certificates valid
- [ ] DNS records configured
- [ ] Load balancer configured (if applicable)
- [ ] CDN configured (if applicable)
- [ ] Health check endpoints working

### Monitoring & Observability

- [ ] Application monitoring configured (Sentry, etc.)
- [ ] Log aggregation set up
- [ ] Metrics collection enabled
- [ ] Alerting rules configured
- [ ] Uptime monitoring active

### Security

- [ ] All secrets rotated
- [ ] CORS configured properly
- [ ] Rate limiting enabled
- [ ] Security headers configured
- [ ] Database credentials use least privilege
- [ ] OAuth apps configured for production domains

## 🚀 Deployment Strategies

### Strategy 1: Blue-Green Deployment

**Concept**: Run two identical environments (blue and green), switch traffic between them.

**Benefits**:

- Zero downtime
- Instant rollback
- Full testing before cutover

**Process**:

```bash
# 1. Deploy to green environment (while blue serves traffic)
docker compose -f docker-compose.green.yml up -d --build

# 2. Run health checks on green
curl https://green.api.yourdomain.com/health

# 3. Run smoke tests
pnpm test:smoke --host=green.api.yourdomain.com

# 4. Switch load balancer from blue to green
# (This is platform-specific)

# 5. Monitor for issues
docker compose -f docker-compose.green.yml logs -f

# 6. If all good, blue becomes the next staging environment
```

**Rollback**:

```bash
# Switch load balancer back to blue
# Takes effect immediately
```

### Strategy 2: Rolling Deployment

**Concept**: Update instances one at a time, maintaining availability.

**Benefits**:

- Gradual rollout
- Detect issues early
- Resource-efficient

**Process**:

```bash
# Using Docker Swarm
docker service update \
  --image izri-api:v2.0.0 \
  --update-parallelism 1 \
  --update-delay 30s \
  izri-api

# Or manually with docker compose
for i in {1..3}; do
  docker compose up -d --no-deps --build api-$i
  sleep 30
  # Check health
  curl http://api-$i:4000/health
done
```

**Rollback**:

```bash
# Roll back to previous image
docker service update \
  --rollback \
  izri-api
```

### Strategy 3: Canary Deployment

**Concept**: Route small percentage of traffic to new version.

**Benefits**:

- Minimal risk
- Real user feedback
- Gradual rollout

**Process**:

```bash
# 1. Deploy canary version
docker compose -f docker-compose.canary.yml up -d

# 2. Configure load balancer: 95% to stable, 5% to canary
# (Platform-specific configuration)

# 3. Monitor metrics
# Compare error rates, latency, etc.

# 4. If metrics good, increase canary traffic
# 95% → 90% → 75% → 50% → 0% (all canary)

# 5. Promote canary to stable
```

**Rollback**:

```bash
# Route all traffic back to stable
# Remove canary containers
docker compose -f docker-compose.canary.yml down
```

## 🛠️ Platform-Specific Deployments

### Option 1: Docker Compose (VPS)

**Setup** (DigitalOcean, AWS EC2, etc.):

```bash
# 1. SSH into server
ssh user@your-server.com

# 2. Clone repository
git clone https://github.com/yourorg/izri.git
cd izri

# 3. Create .env file
nano .env
# Add production environment variables

# 4. Build images
docker compose build

# 5. Start services
docker compose up -d

# 6. Run migrations
docker compose exec api pnpm db:migrate

# 7. Check health
curl http://localhost:4000/health
```

**Update deployment**:

```bash
# 1. Pull latest code
git pull origin main

# 2. Rebuild changed services
docker compose up -d --build

# 3. Check logs
docker compose logs -f
```

### Option 2: Railway

Izri's actual production target. See [`railway.md`](./railway.md) for the per-service `railway.toml` files, env vars, the tag-triggered GitHub Actions workflow, and rollback procedures.

### Option 3: Render

**Configuration** (`render.yaml`):

```yaml
services:
  # API Service
  - type: web
    name: izri-api
    env: docker
    dockerfilePath: ./docker/api/Dockerfile
    dockerContext: .
    envVars:
      - key: DATABASE_URL
        fromDatabase:
          name: postgres
          property: connectionString
      - key: REDIS_URL
        fromService:
          name: redis
          type: redis
          property: connectionString
      - key: NODE_ENV
        value: production
      - key: GITHUB_CLIENT_SECRET
        sync: false  # Set manually
      - key: OPENAI_API_KEY
        sync: false
    healthCheckPath: /health

  # Web Service
  - type: web
    name: izri-web
    env: docker
    dockerfilePath: ./docker/web/Dockerfile
    dockerContext: .
    envVars:
      - key: VITE_API_URL
        value: https://izri-api.onrender.com

databases:
  - name: postgres
    databaseName: izri
    user: izri

  - name: redis
    type: redis
```

**Deploy**:

```bash
# Push to GitHub
git push origin main

# Render auto-deploys on push (if configured)
# Or manually trigger via dashboard
```

### Option 4: Fly.io

**Setup**:

```bash
# 1. Install Fly CLI
curl -L https://fly.io/install.sh | sh

# 2. Login
fly auth login

# 3. Launch app
fly launch
```

**Configuration** (`fly.toml`):

```toml
app = "izri-api"
primary_region = "iad"

[build]
  dockerfile = "docker/api/Dockerfile"

[env]
  NODE_ENV = "production"
  PORT = "4000"

[[services]]
  internal_port = 4000
  protocol = "tcp"

  [[services.ports]]
    port = 80
    handlers = ["http"]

  [[services.ports]]
    port = 443
    handlers = ["tls", "http"]

  [[services.http_checks]]
    interval = "30s"
    timeout = "2s"
    method = "get"
    path = "/health"

[deploy]
  strategy = "rolling"
```

**Deploy**:

```bash
# Deploy
fly deploy

# Set secrets
fly secrets set \
  DATABASE_URL="postgresql://..." \
  GITHUB_CLIENT_SECRET="..." \
  OPENAI_API_KEY="sk-..."

# View logs
fly logs

# SSH into instance
fly ssh console
```

### Option 5: Kubernetes

**Deployment** (`k8s/deployment.yaml`):

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: izri-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: izri-api
  template:
    metadata:
      labels:
        app: izri-api
    spec:
      containers:
      - name: api
        image: ghcr.io/yourorg/izri-api:latest
        ports:
        - containerPort: 4000
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: izri-secrets
              key: database-url
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health
            port: 4000
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health
            port: 4000
          initialDelaySeconds: 5
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: izri-api
spec:
  selector:
    app: izri-api
  ports:
  - port: 80
    targetPort: 4000
  type: LoadBalancer
```

**Deploy**:

```bash
# Apply configuration
kubectl apply -f k8s/

# Check status
kubectl get pods
kubectl get services

# View logs
kubectl logs -f deployment/izri-api

# Scale
kubectl scale deployment izri-api --replicas=5
```

## 📊 Database Migrations

### Pre-Migration Checks

```bash
# 1. Backup database
docker compose exec postgres pg_dump -U postgres izri > backup.sql

# Or for managed databases
pg_dump $DATABASE_URL > backup-$(date +%Y%m%d-%H%M%S).sql

# 2. Verify backup
psql $DATABASE_URL < backup.sql --dry-run
```

### Running Migrations

**Development/staging first**:

```bash
# Test in staging
pnpm db:migrate

# Verify application works
pnpm test:integration
```

**Production migration**:

```bash
# Option 1: Via API container
docker compose exec api pnpm db:migrate

# Option 2: Standalone script
docker run --rm \
  -e DATABASE_URL=$DATABASE_URL \
  izri-api \
  pnpm db:migrate

# Option 3: Platform-specific
railway run pnpm db:migrate
fly ssh console -C "pnpm db:migrate"
```

### Rolling Back Migrations

```bash
# If using Drizzle with down migrations
pnpm db:rollback

# Or restore from backup
psql $DATABASE_URL < backup.sql
```

## 🔄 Rollback Procedures

### Quick Rollback (Docker)

```bash
# 1. Identify previous working image
docker images izri-api

# 2. Tag and deploy
docker tag izri-api:v1.2.3 izri-api:latest
docker compose up -d --no-deps api

# 3. Monitor
docker compose logs -f api
```

### Git-Based Rollback

```bash
# 1. Find last working commit
git log --oneline

# 2. Revert to that commit
git revert <commit-hash>

# 3. Deploy
git push origin main
# Triggers CI/CD deployment

# Or manual
docker compose up -d --build
```

### Database Rollback

```bash
# 1. Stop application
docker compose stop api web

# 2. Restore database
psql $DATABASE_URL < backup-before-migration.sql

# 3. Deploy previous app version
docker compose up -d

# 4. Verify
curl http://localhost:4000/health
```

## 🏥 Health Checks & Monitoring

### Health Check Endpoint

**Implementation** (`src/routes/health.ts`):

```typescript
import { Hono } from 'hono'
import { db } from '@izri/database'
import { redis } from '../lib/redis'

const health = new Hono()

health.get('/health', async (c) => {
  const checks = {
    status: 'ok',
    timestamp: new Date().toISOString(),
    uptime: process.uptime(),
    database: 'unknown',
    redis: 'unknown',
  }

  try {
    // Check database
    await db.execute('SELECT 1')
    checks.database = 'ok'
  } catch (error) {
    checks.database = 'error'
    checks.status = 'degraded'
  }

  try {
    // Check Redis
    await redis.ping()
    checks.redis = 'ok'
  } catch (error) {
    checks.redis = 'error'
    checks.status = 'degraded'
  }

  const statusCode = checks.status === 'ok' ? 200 : 503
  return c.json(checks, statusCode)
})

export { health }
```

### Readiness vs Liveness

**Readiness**: Can the service handle requests?

```typescript
health.get('/ready', async (c) => {
  // More thorough checks
  const ready = 
    await checkDatabase() &&
    await checkRedis() &&
    await checkExternalAPIs()
  
  return c.json({ ready }, ready ? 200 : 503)
})
```

**Liveness**: Is the service running?

```typescript
health.get('/live', async (c) => {
  // Quick check - just respond
  return c.json({ alive: true })
})
```

## 📈 Performance Optimization

### 1. Build Optimization

```dockerfile
# Use buildkit for faster builds
# DOCKER_BUILDKIT=1 docker build .

# Multi-stage builds (already implemented)
# Cache dependencies separately
# Only copy what's needed to runtime
```

### 2. Resource Limits

```yaml
# docker-compose.yml
services:
  api:
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 1G
        reservations:
          cpus: '0.5'
          memory: 512M
```

### 3. Connection Pooling

```typescript
// Database connection pool
import { drizzle } from 'drizzle-orm/node-postgres'
import pg from 'pg'

const pool = new pg.Pool({
  connectionString: env.DATABASE_URL,
  max: 20,                    // Maximum connections
  idleTimeoutMillis: 30000,   // Close idle connections
  connectionTimeoutMillis: 2000,
})

export const db = drizzle(pool)
```

### 4. Caching Strategy

```typescript
import { Redis } from 'ioredis'

const redis = new Redis(env.REDIS_URL)

// Cache database queries
async function getProject(id: string) {
  const cacheKey = `project:${id}`
  
  // Check cache
  const cached = await redis.get(cacheKey)
  if (cached) return JSON.parse(cached)
  
  // Query database
  const project = await db.query.projects.findFirst({
    where: eq(projects.id, id)
  })
  
  // Store in cache
  await redis.setex(cacheKey, 300, JSON.stringify(project))
  
  return project
}
```

## 🔗 Related Documentation

- **Docker**: [Docker configuration](./docker.md)
- **Docker Compose**: [Multi-service setup](./docker-compose.md)
- **Environment**: [Environment variables](./environment-setup.md)
- **Monitoring**: [Monitoring and logging](./monitoring.md)

---

*Need monitoring setup? Continue with [Monitoring & Logging](./monitoring.md) →*