# Monitoring & Logging

> Application monitoring, logging, and observability for production

## 📋 Overview

Effective monitoring and logging are essential for maintaining production systems. This guide covers logging strategies, error tracking, performance monitoring, and alerting.

## 📝 Logging Strategy

### Log Levels

| Level | Usage | Example |
|-------|-------|---------|
| **debug** | Development details | "Query executed in 15ms" |
| **info** | Normal operations | "User logged in", "Server started" |
| **warn** | Warning conditions | "Cache miss", "Deprecated API used" |
| **error** | Error conditions | "Database connection failed" |
| **fatal** | Critical failures | "Out of memory", "Cannot start" |

### Pino Logger Configuration

**Setup** (`src/lib/logger.ts`):

```typescript
import pino from 'pino'

const isProduction = process.env.NODE_ENV === 'production'

export const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  
  // Development: pretty print
  transport: isProduction ? undefined : {
    target: 'pino-pretty',
    options: {
      colorize: true,
      translateTime: 'HH:MM:ss',
      ignore: 'pid,hostname',
    }
  },
  
  // Production: JSON output
  formatters: isProduction ? {
    level: (label) => ({ level: label }),
    bindings: (bindings) => ({
      pid: bindings.pid,
      host: bindings.hostname,
    }),
  } : undefined,
  
  // Add custom fields
  base: {
    env: process.env.NODE_ENV,
    service: 'izri-api',
  },
  
  // Redact sensitive data
  redact: {
    paths: [
      'req.headers.authorization',
      'req.headers.cookie',
      'password',
      'token',
      'apiKey',
    ],
    remove: true,
  },
})
```

### Request Logging Middleware

```typescript
import { Hono } from 'hono'
import { logger as pinoLogger } from 'hono-pino'
import { logger } from './lib/logger'

const app = new Hono()

app.use('*', pinoLogger({
  pino: logger,
  http: {
    // Customize request logging
    reqId: () => crypto.randomUUID(),
    customSuccessMessage: (ctx) => {
      return `${ctx.req.method} ${ctx.req.path} - ${ctx.res.status}`
    },
    customErrorMessage: (ctx, error) => {
      return `${ctx.req.method} ${ctx.req.path} - Error: ${error.message}`
    },
  },
}))
```

### Structured Logging Examples

```typescript
// Basic logging
logger.info('User logged in')

// With context
logger.info({ userId: user.id, email: user.email }, 'User logged in')

// Error logging
logger.error({ err: error, userId: user.id }, 'Failed to create project')

// Performance tracking
const start = Date.now()
const result = await heavyOperation()
logger.info({ duration: Date.now() - start }, 'Heavy operation completed')

// Child loggers (add context)
const reqLogger = logger.child({ requestId: ctx.get('requestId') })
reqLogger.info('Processing request')
```

## 🔍 Error Tracking

### Sentry Integration

**Installation**:

```bash
pnpm add @sentry/node @sentry/tracing
```

**Configuration** (`src/lib/sentry.ts`):

```typescript
import * as Sentry from '@sentry/node'
import { ProfilingIntegration } from '@sentry/profiling-node'

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  environment: process.env.NODE_ENV,
  
  // Performance monitoring
  tracesSampleRate: 0.1,  // 10% of transactions
  
  // Profiling
  profilesSampleRate: 0.1,
  integrations: [
    new ProfilingIntegration(),
  ],
  
  // Filter out health checks
  beforeSend(event, hint) {
    const url = event.request?.url
    if (url?.includes('/health') || url?.includes('/metrics')) {
      return null
    }
    return event
  },
})
```

**Hono Integration**:

```typescript
import { Hono } from 'hono'
import * as Sentry from '@sentry/node'

const app = new Hono()

// Request context
app.use('*', async (c, next) => {
  const transaction = Sentry.startTransaction({
    op: 'http.server',
    name: `${c.req.method} ${c.req.path}`,
  })
  
  Sentry.configureScope((scope) => {
    scope.setSpan(transaction)
    scope.setContext('request', {
      method: c.req.method,
      url: c.req.url,
      headers: Object.fromEntries(c.req.raw.headers),
    })
  })
  
  try {
    await next()
  } finally {
    transaction.finish()
  }
})

// Error handler
app.onError((err, c) => {
  Sentry.captureException(err, {
    contexts: {
      request: {
        method: c.req.method,
        url: c.req.url,
      },
      user: {
        id: c.get('userId'),
      },
    },
  })
  
  logger.error({ err }, 'Unhandled error')
  return c.json({ error: 'Internal server error' }, 500)
})
```

**Manual Error Tracking**:

```typescript
try {
  await riskyOperation()
} catch (error) {
  // Add custom context
  Sentry.withScope((scope) => {
    scope.setLevel('error')
    scope.setTag('operation', 'createProject')
    scope.setContext('details', {
      projectId,
      userId,
    })
    Sentry.captureException(error)
  })
  
  throw error
}
```

## 📊 Metrics Collection

### OpenTelemetry Setup

**Installation**:

```bash
pnpm add @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node
```

**Configuration** (`src/lib/telemetry.ts`):

```typescript
import { NodeSDK } from '@opentelemetry/sdk-node'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'

const sdk = new NodeSDK({
  serviceName: 'izri-api',
  
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT + '/v1/traces',
  }),
  
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT + '/v1/metrics',
    }),
    exportIntervalMillis: 60000,  // Export every minute
  }),
  
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': {
        enabled: false,  // Too noisy
      },
    }),
  ],
})

sdk.start()

// Graceful shutdown
process.on('SIGTERM', () => {
  sdk.shutdown().finally(() => process.exit(0))
})
```

**Custom Metrics**:

```typescript
import { metrics } from '@opentelemetry/api'

const meter = metrics.getMeter('izri-api')

// Counter
const requestCounter = meter.createCounter('http_requests_total', {
  description: 'Total number of HTTP requests',
})

requestCounter.add(1, { method: 'GET', route: '/api/projects' })

// Histogram (for latencies)
const requestDuration = meter.createHistogram('http_request_duration_ms', {
  description: 'HTTP request duration in milliseconds',
})

const start = Date.now()
await handleRequest()
requestDuration.record(Date.now() - start, { method: 'GET', route: '/api/projects' })

// Gauge (current values)
const activeConnections = meter.createObservableGauge('active_connections', {
  description: 'Number of active connections',
})

activeConnections.addCallback((result) => {
  result.observe(getActiveConnectionCount())
})
```

### Custom Metrics Endpoint

```typescript
import { Hono } from 'hono'

const metrics = new Hono()

let requestCount = 0
let errorCount = 0
const startTime = Date.now()

// Middleware to track metrics
app.use('*', async (c, next) => {
  requestCount++
  try {
    await next()
    if (c.res.status >= 400) errorCount++
  } catch (error) {
    errorCount++
    throw error
  }
})

// Prometheus-style metrics endpoint
metrics.get('/metrics', (c) => {
  const uptime = (Date.now() - startTime) / 1000
  
  return c.text(`
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total ${requestCount}

# HELP http_errors_total Total HTTP errors
# TYPE http_errors_total counter
http_errors_total ${errorCount}

# HELP process_uptime_seconds Process uptime in seconds
# TYPE process_uptime_seconds gauge
process_uptime_seconds ${uptime}

# HELP nodejs_memory_usage_bytes Node.js memory usage
# TYPE nodejs_memory_usage_bytes gauge
nodejs_memory_usage_bytes{type="heapUsed"} ${process.memoryUsage().heapUsed}
nodejs_memory_usage_bytes{type="heapTotal"} ${process.memoryUsage().heapTotal}
nodejs_memory_usage_bytes{type="rss"} ${process.memoryUsage().rss}
`.trim())
})
```

## 📈 Application Performance Monitoring (APM)

### Custom Performance Tracking

```typescript
import { performance } from 'perf_hooks'

class PerformanceTracker {
  private marks = new Map<string, number>()
  
  start(name: string) {
    this.marks.set(name, performance.now())
  }
  
  end(name: string) {
    const start = this.marks.get(name)
    if (!start) return
    
    const duration = performance.now() - start
    logger.info({ name, duration }, 'Performance measurement')
    
    // Send to APM
    if (process.env.OTEL_EXPORTER_OTLP_ENDPOINT) {
      // OpenTelemetry will pick this up
    }
    
    this.marks.delete(name)
    return duration
  }
}

export const perf = new PerformanceTracker()

// Usage
perf.start('database.query.projects')
const projects = await db.query.projects.findMany()
perf.end('database.query.projects')
```

### Database Query Monitoring

```typescript
import { drizzle } from 'drizzle-orm/node-postgres'
import { logger } from './logger'

export const db = drizzle(pool, {
  logger: {
    logQuery(query, params) {
      logger.debug({ query, params }, 'Database query')
    },
  },
})

// Slow query detection
const SLOW_QUERY_THRESHOLD = 1000 // 1 second

export async function monitoredQuery<T>(
  name: string,
  query: () => Promise<T>
): Promise<T> {
  const start = Date.now()
  try {
    const result = await query()
    const duration = Date.now() - start
    
    if (duration > SLOW_QUERY_THRESHOLD) {
      logger.warn({ name, duration }, 'Slow query detected')
    }
    
    return result
  } catch (error) {
    logger.error({ name, err: error }, 'Query failed')
    throw error
  }
}

// Usage
const projects = await monitoredQuery(
  'getProjectsByUserId',
  () => db.query.projects.findMany({ where: eq(projects.userId, userId) })
)
```

## 🚨 Alerting

### Alert Configuration Examples

**Uptime monitoring** (UptimeRobot, Pingdom, etc.):

```yaml
monitors:
  - name: API Health Check
    url: https://api.yourdomain.com/health
    interval: 60  # seconds
    timeout: 30
    expected_status: 200
    
  - name: Web Application
    url: https://yourdomain.com
    interval: 60
    timeout: 30
```

**Error rate alerts** (Sentry):

```javascript
// Sentry Alert Rules
{
  conditions: [
    {
      id: 'sentry.rules.conditions.event_frequency.EventFrequencyCondition',
      interval: '5m',
      value: 100,  // More than 100 errors in 5 minutes
    }
  ],
  actions: [
    {
      id: 'sentry.mail.actions.NotifyEmailAction',
      targetType: 'Team',
    },
    {
      id: 'sentry.integrations.slack.notify_action.SlackNotifyServiceAction',
      channel: '#alerts',
    }
  ]
}
```

**Custom alerts** (via metrics):

```typescript
import { WebClient } from '@slack/web-api'

const slack = new WebClient(process.env.SLACK_BOT_TOKEN)

async function checkErrorRate() {
  const errorRate = errorCount / requestCount
  
  if (errorRate > 0.05) {  // 5% error rate
    await slack.chat.postMessage({
      channel: '#alerts',
      text: `🚨 High error rate detected: ${(errorRate * 100).toFixed(2)}%`,
      blocks: [
        {
          type: 'section',
          text: {
            type: 'mrkdwn',
            text: `*Error Rate Alert*\n\nCurrent error rate: ${(errorRate * 100).toFixed(2)}%\nThreshold: 5%\nTotal requests: ${requestCount}\nTotal errors: ${errorCount}`,
          },
        },
      ],
    })
  }
}

// Run every minute
setInterval(checkErrorRate, 60000)
```

## 📦 Log Aggregation

### Using Docker Logs

```bash
# View all logs
docker compose logs -f

# Specific service
docker compose logs -f api

# Since timestamp
docker compose logs --since 2024-01-01T10:00:00

# Tail last 100 lines
docker compose logs --tail=100 api
```

### Centralized Logging (ELK Stack)

**Docker Compose with Filebeat**:

```yaml
services:
  api:
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
  
  filebeat:
    image: docker.elastic.co/beats/filebeat:8.11.0
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
```

**Filebeat configuration** (`filebeat.yml`):

```yaml
filebeat.inputs:
  - type: container
    paths:
      - '/var/lib/docker/containers/*/*.log'
    processors:
      - add_docker_metadata:
          host: "unix:///var/run/docker.sock"

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  
# Or send to Logstash
output.logstash:
  hosts: ["logstash:5044"]
```

### Cloud Logging

**AWS CloudWatch**:

```yaml
# docker-compose.yml
services:
  api:
    logging:
      driver: awslogs
      options:
        awslogs-region: us-east-1
        awslogs-group: /izri/api
        awslogs-stream: api
```

**Google Cloud Logging**:

```typescript
import { Logging } from '@google-cloud/logging'

const logging = new Logging()
const log = logging.log('izri-api')

function cloudLog(message: string, severity: string = 'INFO') {
  const entry = log.entry({
    severity,
    resource: {
      type: 'global',
    },
  }, message)
  
  log.write(entry)
}
```

## 🎯 Monitoring Best Practices

### 1. Log Levels in Production

```typescript
// ❌ Don't log everything
logger.debug('Variable value:', someVar)
logger.debug('Entering function')

// ✅ Use appropriate levels
logger.info({ userId, projectId }, 'Project created')
logger.warn({ duration }, 'Slow query detected')
logger.error({ err }, 'Failed to process request')
```

### 2. Structured Logging

```typescript
// ❌ Unstructured
logger.info('User john@example.com created project MyProject with ID 123')

// ✅ Structured
logger.info({
  userId: 'user_123',
  email: 'john@example.com',
  projectId: 'proj_123',
  projectName: 'MyProject',
}, 'Project created')
```

### 3. Correlation IDs

```typescript
// Add request ID to all logs for a request
app.use('*', async (c, next) => {
  const requestId = crypto.randomUUID()
  c.set('requestId', requestId)
  
  // Child logger with requestId
  c.set('logger', logger.child({ requestId }))
  
  await next()
})

// Use in handlers
app.get('/projects', async (c) => {
  const log = c.get('logger')
  log.info('Fetching projects')  // Includes requestId
})
```

### 4. Sensitive Data Redaction

```typescript
// Configure in logger
redact: {
  paths: [
    'password',
    'token',
    'apiKey',
    'authorization',
    'cookie',
  ],
  remove: true,
}

// Or manually
function sanitize(obj: any) {
  const clean = { ...obj }
  delete clean.password
  delete clean.token
  return clean
}

logger.info(sanitize(user), 'User logged in')
```

## 🔗 Related Documentation

- **Docker**: [Docker configuration](./docker.md)
- **Production**: [Production deployment](./production.md)
- **Backend**: [Backend logging](../backend/logging.md)
- **Error Handling**: [Error handling patterns](../backend/error-handling.md)

---

*Deployment section complete! Continue with [Features Documentation](../features/README.md) →*