Data Flow

Understanding how data moves through the Izri system

📋 Overview

This document explains how data flows through different layers of the application, from user interaction to database storage and back. Understanding these patterns is crucial for debugging, optimization, and feature development.

🔄 Request/Response Lifecycle

Complete User Request Flow

sequenceDiagram
    participant U as User Browser
    participant R as React Router
    participant TQ as TanStack Query
    participant TC as tRPC Client
    participant H as Hono Server
    participant TS as tRPC Server
    participant M as Middleware
    participant P as Procedure
    participant DB as Database
    participant Cache as Redis
    
    U->>R: Navigate to /projects
    R->>R: Load route component
    R->>TQ: useQuery('projects.list')
    TQ->>TC: trpc.projects.list.query()
    TC->>H: POST /trpc/projects.list
    H->>M: Auth middleware
    M->>M: Verify session
    M->>TS: Forward with context
    TS->>P: Execute procedure
    P->>Cache: Check cache
    alt Cache Hit
        Cache-->>P: Return cached data
    else Cache Miss
        P->>DB: SELECT * FROM projects
        DB-->>P: Return rows
        P->>Cache: Store in cache
    end
    P-->>TS: Return data
    TS-->>H: JSON response
    H-->>TC: HTTP 200 + JSON
    TC-->>TQ: Parsed data
    TQ-->>R: Update component
    R-->>U: Render UI

Step-by-Step Breakdown

1. User Interaction

// User clicks "View Projects"
// Route: /projects

2. Route Component Loads

// app/routes/projects/_index.tsx
import { useQuery } from '@tanstack/react-query'
import { trpc } from '@/lib/trpc'

export default function Projects() {
  const { data, isLoading } = trpc.projects.list.useQuery()
  
  if (isLoading) return <Spinner />
  return <ProjectList projects={data} />
}

3. tRPC Client Request

// @/lib/trpc.ts creates the client
export const trpc = createTRPCReact<AppRouter>()

// Request:
POST http://localhost:4000/trpc/projects.list
Content-Type: application/json
Cookie: session=...

{}  // Empty input for this query

4. Hono Server Receives Request

// apps/api/src/index.ts
app.use('*', logger())
app.use('*', cors())
app.use('*', session())  // Auth middleware

app.use('/trpc/*', trpcMiddleware({
  router: appRouter,
  createContext
}))

5. Context Creation

// packages/trpc/src/context.ts
export async function createContext({ req, res }) {
  const session = await getSession(req)
  
  return {
    db,           // Database client
    redis,        // Cache client
    userId: session?.userId,
    req,
    res
  }
}

6. Procedure Execution

// packages/trpc/src/routers/projects.ts
export const projectRouter = router({
  list: protectedProcedure
    .query(async ({ ctx }) => {
      // Check cache first
      const cacheKey = `projects:${ctx.userId}`
      const cached = await ctx.redis.get(cacheKey)
      
      if (cached) {
        return JSON.parse(cached)
      }
      
      // Query database
      const projects = await ctx.db
        .select()
        .from(projects)
        .where(eq(projects.userId, ctx.userId))
      
      // Store in cache (5 minutes)
      await ctx.redis.setex(cacheKey, 300, JSON.stringify(projects))
      
      return projects
    })
})

7. Database Query

// Drizzle translates to SQL:
SELECT * FROM projects WHERE user_id = $1

// PostgreSQL executes and returns rows

8. Response Back to Client

// tRPC formats response:
{
  "result": {
    "data": [
      { "id": "...", "name": "Project 1", ... },
      { "id": "...", "name": "Project 2", ... }
    ]
  }
}

// TanStack Query updates cache
// React re-renders with data

📝 Mutation Flow (Write Operations)

Create Project Example

sequenceDiagram
    participant U as User
    participant F as Form Component
    participant TC as tRPC Client
    participant API as API Server
    participant Val as Validation
    participant DB as Database
    participant Cache as Redis
    
    U->>F: Fill form & submit
    F->>F: Client-side validation
    F->>TC: trpc.projects.create.mutate(data)
    TC->>API: POST /trpc/projects.create
    API->>Val: Validate with Zod
    alt Invalid Input
        Val-->>TC: TRPCError (400)
        TC-->>F: Show error
    else Valid Input
        Val->>DB: BEGIN TRANSACTION
        DB->>DB: INSERT INTO projects
        DB->>DB: INSERT INTO project_settings
        DB->>DB: COMMIT
        DB-->>API: Return created project
        API->>Cache: Invalidate cache
        Cache-->>API: OK
        API-->>TC: Success response
        TC->>TC: Invalidate queries
        TC-->>F: onSuccess callback
        F-->>U: Show success + redirect
    end

Code Example

Frontend (Form):

// app/components/CreateProjectForm.tsx
import { trpc } from '@/lib/trpc'

export function CreateProjectForm() {
  const utils = trpc.useUtils()
  
  const createProject = trpc.projects.create.useMutation({
    onSuccess: (data) => {
      // Invalidate and refetch projects list
      utils.projects.list.invalidate()
      
      // Navigate to new project
      navigate(`/projects/${data.id}`)
    },
    onError: (error) => {
      toast.error(error.message)
    }
  })
  
  const handleSubmit = (values) => {
    createProject.mutate(values)
  }
  
  return <form onSubmit={handleSubmit}>...</form>
}

Backend (Procedure):

// packages/trpc/src/routers/projects.ts
import { z } from 'zod'

const createProjectSchema = z.object({
  name: z.string().min(1).max(255),
  repository: z.string().url(),
  branch: z.string().default('main'),
  language: z.string(),
  framework: z.string().optional(),
})

export const projectRouter = router({
  create: protectedProcedure
    .input(createProjectSchema)
    .mutation(async ({ ctx, input }) => {
      // Start transaction
      return await ctx.db.transaction(async (tx) => {
        // Create project
        const [project] = await tx
          .insert(projects)
          .values({
            id: ulid(), // ULID for database IDs
            userId: ctx.userId,
            ...input,
          })
          .returning()
        
        // Create default settings
        await tx.insert(projectSettings).values({
          id: ulid(), // ULID for database IDs
          projectId: project.id,
          autoRunTests: true,
          notifyOnFailure: true,
        })
        
        // Invalidate cache
        await ctx.redis.del(`projects:${ctx.userId}`)
        
        return project
      })
    })
})

🔐 Authentication Flow

Login with GitHub OAuth

sequenceDiagram
    participant U as User
    participant W as Web App
    participant API as API Server
    participant GH as GitHub
    participant DB as Database
    participant Redis as Redis
    
    U->>W: Click "Login with GitHub"
    W->>GH: Redirect to OAuth
    GH->>U: Show authorization
    U->>GH: Approve
    GH->>W: Redirect with code
    W->>API: POST /auth/callback/github
    API->>GH: Exchange code for token
    GH-->>API: Access token + user data
    API->>DB: Upsert user
    DB-->>API: User record
    API->>Redis: Create session
    Redis-->>API: Session ID
    API-->>W: Set-Cookie: session=...
    W-->>U: Redirect to dashboard

Session Validation (Every Request)

sequenceDiagram
    participant C as Client
    participant M as Auth Middleware
    participant Redis as Redis
    participant DB as Database
    
    C->>M: Request with Cookie
    M->>Redis: GET session:{sessionId}
    alt Session Valid
        Redis-->>M: User ID
        M->>DB: Get user data (optional)
        DB-->>M: User record
        M-->>M: Add to context
    else Session Invalid/Expired
        Redis-->>M: null
        M-->>C: 401 Unauthorized
    end

📊 Database Access Patterns

Read Operations

Single Record:

const project = await db
  .select()
  .from(projects)
  .where(eq(projects.id, projectId))
  .limit(1)
  .then(rows => rows[0])

List with Pagination:

const projectsList = await db
  .select()
  .from(projects)
  .where(eq(projects.userId, userId))
  .orderBy(desc(projects.createdAt))
  .limit(20)
  .offset((page - 1) * 20)

With Relations (Join):

const projectsWithRuns = await db
  .select({
    project: projects,
    testRun: testRuns,
  })
  .from(projects)
  .leftJoin(testRuns, eq(testRuns.projectId, projects.id))
  .where(eq(projects.userId, userId))

Aggregations:

import { count, avg } from 'drizzle-orm'

const stats = await db
  .select({
    totalProjects: count(projects.id),
    avgCoverage: avg(testRuns.coverage),
  })
  .from(projects)
  .leftJoin(testRuns, eq(testRuns.projectId, projects.id))
  .where(eq(projects.userId, userId))

Write Operations

Insert:

import { ulid } from 'ulid'

const [project] = await db
  .insert(projects)
  .values({
    id: ulid(), // ULID for database IDs
    name: 'New Project',
    userId,
  })
  .returning()

Update:

await db
  .update(projects)
  .set({
    name: 'Updated Name',
    updatedAt: new Date(),
  })
  .where(eq(projects.id, projectId))

Delete:

await db
  .delete(projects)
  .where(eq(projects.id, projectId))

Transaction (Multiple Operations):

await db.transaction(async (tx) => {
  // Delete test results
  await tx
    .delete(testResults)
    .where(eq(testResults.testRunId, runId))
  
  // Delete test run
  await tx
    .delete(testRuns)
    .where(eq(testRuns.id, runId))
})

💾 Caching Strategy

Cache Layers

┌─────────────────────┐
│   Browser Cache     │ (React Query)
└──────────┬──────────┘
           │
┌──────────▼──────────┐
│   Redis Cache       │ (Server-side)
└──────────┬──────────┘
           │
┌──────────▼──────────┐
│   PostgreSQL        │ (Source of truth)
└─────────────────────┘

React Query (Browser)

// Automatic caching with TanStack Query
const { data } = trpc.projects.list.useQuery(undefined, {
  staleTime: 5 * 60 * 1000,  // 5 minutes
  cacheTime: 10 * 60 * 1000, // 10 minutes
})

// Invalidate when data changes
const utils = trpc.useUtils()
utils.projects.list.invalidate()

Redis (Server)

// Cache read
const cacheKey = `projects:${userId}`
const cached = await redis.get(cacheKey)
if (cached) {
  return JSON.parse(cached)
}

// Cache write
const data = await db.select().from(projects)
await redis.setex(cacheKey, 300, JSON.stringify(data))  // 5 min TTL

// Cache invalidation
await redis.del(`projects:${userId}`)

// Pattern-based invalidation
await redis.del(`projects:${userId}:*`)

Cache Invalidation Strategy

On Create:

// Invalidate list cache
await redis.del(`projects:${userId}`)

On Update:

// Invalidate specific item and list
await redis.del(`project:${projectId}`)
await redis.del(`projects:${userId}`)

On Delete:

// Invalidate everything related
await redis.del(`project:${projectId}`)
await redis.del(`projects:${userId}`)
await redis.del(`project:${projectId}:runs`)

🎯 Error Handling Flow

Error Propagation

graph TD
    A[Error Occurs] --> B{Where?}
    B -->|Database| C[Drizzle Error]
    B -->|Validation| D[Zod Error]
    B -->|Business Logic| E[Custom Error]
    
    C --> F[tRPC Error Handler]
    D --> F
    E --> F
    
    F --> G[Format Error]
    G --> H[HTTP Response]
    H --> I[tRPC Client]
    I --> J[React Query]
    J --> K[UI Error Display]

Error Types

Validation Error:

// Input validation fails
{
  "error": {
    "code": "BAD_REQUEST",
    "message": "Validation error",
    "data": {
      "zodError": {
        "fieldErrors": {
          "name": ["String must contain at least 1 character(s)"]
        }
      }
    }
  }
}

Authentication Error:

{
  "error": {
    "code": "UNAUTHORIZED",
    "message": "Not authenticated"
  }
}

Database Error:

{
  "error": {
    "code": "INTERNAL_SERVER_ERROR",
    "message": "Database error occurred"
  }
}

Handling in Frontend

const createProject = trpc.projects.create.useMutation({
  onError: (error) => {
    if (error.data?.code === 'UNAUTHORIZED') {
      navigate('/login')
    } else if (error.data?.zodError) {
      // Show field-specific errors
      setFieldErrors(error.data.zodError.fieldErrors)
    } else {
      toast.error(error.message)
    }
  }
})

🔗 Related Documentation

• System Overview: High-level architecture
• Technology Stack: Technologies used
• Database Management: Database operations
• Backend Documentation: API implementation details

Need help tracing a specific data flow? Use logging, Drizzle Studio, and browser DevTools to follow requests.