Loops, Error Handling and Incidents

Chapter 12 | Loops, Error Handling and Incidents Written for: developers at every level who want a system that recovers when it fails, and does not fail the same way twice

Deep loop debugging — root cause analysis

Chapter 4 taught the protocol for stopping a loop. This chapter goes further: how to analyse the root cause systematically, and how to design code up front so that loops happen less often.

The 5 Whys framework for loop debugging

The “5 Whys” technique from the Toyota Production System works very well for debugging with Claude. You do not have to ask “why” a full five times every session — you ask until you reach the actual root cause.

Example: fixing a real bug with 5 Whys

# Situation: payments fail intermittently
 
> Why 1: why does the payment fail?
  → Stripe API returns the error "card_declined"
 
> Why 2: why is the card declined?
  → Stripe says insufficient_funds, but the user says the balance is fine
 
> Why 3: why does Stripe see insufficient_funds when the balance is fine?
  → Looking at the request payload, the amount sent is 1000
     but it should be 10.00 (different units: baht vs satang)
 
> Why 4: why is the amount sent in the wrong unit?
  → The code in checkout.ts never converts baht → satang
     before sending to Stripe (Stripe uses the smallest unit of the currency)
 
> Why 5: why is the conversion missing?
  → The developer misread the doc and assumed Stripe uses the major unit,
     and there is no unit test verifying the amount
 
# Root cause: missing unit test + unclear documentation
# Real fix: correct the conversion + add a test + add a note in CLAUDE.md

Prompting Claude to run 5 Whys

> Run a 5 Whys analysis for this bug:
  [describe the symptom]
 
  Do it like this:
  1. Why 1: describe the symptom you can see
  2. Why 2-5: keep asking, each one building on the previous answer
  3. Stop when you reach the real root cause (fewer than 5 is fine)
  4. Separate "fix the symptom" from "fix the root cause"
  5. Propose a solution for the root cause
  6. Propose a way to prevent a recurrence (systematic fix)

Debugging playbook — a guide per bug class

Instead of describing the problem in vague terms, identify the class of bug first, then use the matching playbook:

	Playbook A: Logic bug — wrong result, no error

The hardest kind to debug, because there is no error message to follow. Claude tends to loop most on this class.

> Logic bug analysis:
  What I see: [input] → [actual output]
  What I want: [input] → [expected output]
 
  Do the following:
  1. Trace the execution path: describe step by step what the code does
     from receiving the input to returning the output
 
  2. Find the divergence point: where do actual and expected start to differ?
 
  3. Add assertion checks along the way:
     console.assert(value === expected, `Step N: got ${value}, want ${expected}`)
 
  4. Run it and see which step the assertion fails at
 
  Do not change any code until you can identify the divergence point

	Playbook B: Performance bug — unusually slow

> Performance investigation:
  Symptom: [which part is slow] takes about [X seconds/ms]
  Target: must be faster than [Y seconds/ms]
 
  Follow this order:
 
  Phase 1 — measure before changing anything:
  Add timing in @[file]:
  - console.time("total")
  - console.time("db-query")
  - console.time("data-transform")
  - console.time("response")
  Then run it 3 times and record the average
 
  Phase 2 — identify the bottleneck:
  Which step takes the most time?
 
  Phase 3 — optimise the bottleneck only:
  Do not optimise the parts that are already fast
 
  Phase 4 — measure again:
  Compare before/after; it must hit the [Y ms] target

	Playbook C: Memory leak — RAM creeps up until the system slows

> Memory leak investigation:
  Symptom: memory keeps climbing during long runs
 
  Step 1 — confirm there really is a leak:
  Add memory monitoring:
  setInterval(() => {
    const mem = process.memoryUsage();
    console.log(`RSS: ${Math.round(mem.rss/1024/1024)}MB`,
                `Heap: ${Math.round(mem.heapUsed/1024/1024)}MB`);
  }, 5000);
 
  Step 2 — find the pattern:
  Which action makes memory grow?
  Every request? After an upload? After a WebSocket connect?
 
  Step 3 — analyse @[file] for these patterns:
  - Event listeners never removed with removeEventListener
  - setTimeout/setInterval never cleared with clearTimeout/clearInterval
  - Global variables holding an array/object forever
  - A cache with no eviction policy
  - Closures capturing a large object
 
  Step 4 — fix it, then verify memory is flat after 30 minutes of running

	Playbook D: Concurrency bug — only wrong under high traffic

> Concurrency analysis:
  Symptom: fine normally, but breaks with concurrent users
 
  Step 1 — reproduce locally:
  Write a load test script that sends concurrent requests:
  for (let i = 0; i < 50; i++) {
    promises.push(fetch("/api/[endpoint]"));
  }
  await Promise.all(promises);
 
  Step 2 — analyse shared state:
  Find variables in @[file] that are:
  - module-level (not request-level)
  - read and written in the same handler
  - without any locking/synchronisation
 
  Step 3 — propose a fix:
  - Database-level locking: SELECT FOR UPDATE
  - Optimistic locking: version field + retry
  - Queue: serialise concurrent operations
  - Idempotency key: prevent duplicate processing
 
  Explain the tradeoffs of each option before choosing

Anti-loop patterns — designing so loops happen less

The best way to avoid loops is to design the code and the prompt well from the start:

Pattern 1: Contract-first development

Define the interface and the contract clearly before implementing, so Claude has a precise specification and does not have to guess:

# Instead of saying "build a payment service"
# write the contract first, then have Claude implement it
 
> This is the contract I want:
 
  // Input
  interface ProcessPaymentInput {
    userId: string;
    amount: number;        // unit: baht (e.g. 99.50)
    currency: "THB" | "USD";
    description: string;
    idempotencyKey: string; // prevents duplicate charges
  }
 
  // Success Output
  interface ProcessPaymentSuccess {
    success: true;
    transactionId: string;
    amount: number;         // confirms the amount actually charged
    timestamp: Date;
  }
 
  // Error Output
  interface ProcessPaymentError {
    success: false;
    errorCode: "CARD_DECLINED" | "INSUFFICIENT_FUNDS" |
               "INVALID_CARD" | "PROCESSING_ERROR";
    message: string;       // for logs (not shown to the user)
    userMessage: string;   // for display to the user (in Thai)
    retryable: boolean;    // tells the client whether a retry is possible
  }
 
  implement processPayment(input: ProcessPaymentInput)
  : Promise<ProcessPaymentSuccess | ProcessPaymentError>
  Use Stripe as the payment processor
  Convert amount from baht → satang before sending to Stripe

Pattern 2: Error budget strategy

Decide in advance which classes of error Claude may retry on its own, and which ones require it to stop and ask:

# Add to CLAUDE.md
## 🔄 Error Handling Strategy
 
### Claude may retry on its own (no need to ask):
- TypeScript type error → correct the type
- Wrong import path → fix the path
- Missing await → add await
- Unused variable → remove it
 
### Claude must ask before retrying:
- Logic error (result does not match expectations) → explain your understanding first
- Database schema mismatch → show the schema you see, then ask
- External API error → show the error code + ask about intent
 
### Claude must stop and report:
- Anything security-related → never fix it unilaterally
- Two rounds of fixes with no success → summarise what was tried + ask for more information
- A change needed outside the agreed scope → get approval first

Pattern 3: Checkpoint system

Split the work into small checkpoints and verify each one before moving on. This prevents the snowball effect of accumulating bugs:

> Build a user registration system using checkpoints:
 
  Checkpoint 1: Database schema
  - Create the Prisma schema for User
  - Run the migration
  - ✓ verify: npx prisma db pull and inspect the schema
  STOP → wait for approval before Checkpoint 2
 
  Checkpoint 2: Validation layer
  - Create the Zod schema for the registration input
  - Write unit tests covering valid/invalid cases
  - ✓ verify: npm test must pass in full
  STOP → wait for approval before Checkpoint 3
 
  Checkpoint 3: Service layer
  - Create the registerUser() function
  - Hash the password, check for a duplicate email
  - ✓ verify: integration test against the test database
  STOP → wait for approval before Checkpoint 4
 
  Checkpoint 4: API route
  - Create POST /api/auth/register
  - ✓ verify: curl test for every case
 
  If any checkpoint fails → stop, do not move to the next one

Using Claude to debug Claude

A very powerful technique: have a fresh Claude analyse the problem without knowing which Claude wrote the code.

# Session 1: Claude A builds the code and hits a bug
# Claude A cannot fix it in 2 rounds
 
# Open a new session (Claude B)
> [Fresh session - no prior context]
 
  I was handed this code and it has a bug:
  [paste the whole file]
 
  Symptom: [describe]
  Error: [copy the error]
 
  Analyse this code from the outside:
  1. What is this code trying to do?
  2. Do you see any design issues?
  3. Where is the bug likely to be?
  4. If you were writing this code from scratch,
     what would you change about the approach?
 
# Claude B often sees what Claude A overlooked,
# because it carries no cognitive bias from having written the code

💡 When to use Claude to debug Claude
Use it when: Claude has looped for more than 2 rounds without a fix
Use it when: the code is complex enough to be hard to explain
Use it when: you want a second opinion before merging
Skip it for: easy bugs, or anything fixed on the first try

Production quality — from the first line to going live

Production code does not just mean “it runs”. It means code that is reliable, maintainable, observable and recoverable. This section walks every stage from design to real users.

Stage 1 — Requirements and design

Most work fails at this stage, not at implementation. Always have Claude clarify the requirements first.

# Prompt for clarifying requirements
> Before implementing, I want to clarify the requirements:
 
  Feature wanted: [describe]
 
  Ask me the questions that need answering before implementation:
  1. Happy path: what does the normal flow look like?
  2. Edge cases: which special cases must be handled?
  3. Error cases: what happens if X fails?
  4. Performance: how many concurrent users must it support?
  5. Security: who can access it? What is the data sensitivity level?
  6. Rollback: if this feature breaks, how do we roll it back?
 
  Once clarified, summarise the acceptance criteria
  used to verify that the feature is complete and correct

A design document Claude helps you write

> Create a design document for [Feature] covering:
 
  ## Summary
  Describe the feature in 2-3 sentences
 
  ## Architecture
  - Data flow diagram (text format)
  - Components involved
  - External dependencies
 
  ## API Contract
  - New endpoints (input/output types)
  - Error responses
 
  ## Database Changes
  - Tables/columns added or changed
  - Migration strategy
  - Rollback plan
 
  ## Testing Strategy
  - Unit tests to write
  - Integration tests
  - E2E scenarios
 
  ## Risks & Mitigations
  - Risks you can see
  - How to reduce each one
 
  ## Acceptance Criteria
  ✓ [criterion 1]
  ✓ [criterion 2]

Stage 2 — Implementation with quality gates

Every implementation phase has a quality gate it must pass before moving on. Claude can help verify each gate.

Phase	Work	Quality gate	How Claude helps
Data layer	Schema, migrations, indexes	Migration runs and rolls back	Writes the migration + tests
Business logic	Core functions, validation	Unit tests pass 100%	Writes tests + implements
API layer	Routes, middleware, auth	Integration tests pass	Writes tests + routes
UI layer	Components, forms, states	E2E tests pass on key flows	Writes Playwright tests
Performance	Query optimisation, caching	Load test hits the target	Analyses the bottleneck
Security	Auth checks, input validation	Security checklist passes	Runs the OWASP checklist

Stage 3 — A complete testing strategy

Testing is the part vibe coders skip most often, and it is exactly what separates production code from a prototype.

	Unit testing with Vitest + Claude

# Have Claude write comprehensive unit tests
> Write unit tests for @src/lib/pricing.ts
  covering ALL cases:
 
  1. Happy paths (valid input → expected output)
  2. Edge cases:
     - boundary numbers: 0, negative, MAX_SAFE_INTEGER
     - null/undefined inputs
     - empty arrays/objects
  3. Error cases:
     - invalid input type
     - business rule violations
  4. Precision cases:
     - floating point (0.1 + 0.2 ≠ 0.3)
     - currency rounding
 
  Use Vitest, organised into describe blocks
  Test names must be readable: "should return X when Y"
 
# Example of good output
describe("calculateDiscount", () => {
  describe("percentage discount", () => {
    it("should apply 10% discount to 1000", () => {
      expect(calculateDiscount(1000, { type: "percent", value: 10 }))
        .toBe(900);
    });
    it("should handle 0% discount", () => {...});
    it("should cap at 100% discount", () => {...});
  });
  describe("fixed discount", () => {...});
  describe("edge cases", () => {...});
});

	Integration testing with Supertest

> Write integration tests for POST /api/orders
  using Supertest + a test database
 
  Test cases:
  1. ✅ Success: order created correctly → 201 + order object
  2. ❌ Unauthorized: no token → 401
  3. ❌ Validation: incomplete data → 400 + error details
  4. ❌ Business rule: product out of stock → 422 + user message
  5. ⚡ Idempotency: same request sent twice → only one order created
  6. 🔒 Authorization: user views someone else's order → 403
 
  Setup:
  - Start each test from a clean test database
  - Clear the database after each test
  - Mock the Stripe API (never call the real API in a test)
 
# Example
describe("POST /api/orders", () => {
  beforeEach(async () => {
    await db.cleanDatabase();
    await db.seed.basicProducts();
  });
 
  it("should create order and return 201", async () => {
    const res = await request(app)
      .post("/api/orders")
      .set("Authorization", `Bearer ${testToken}`)
      .send({ items: [{ productId: "p1", quantity: 2 }] });
 
    expect(res.status).toBe(201);
    expect(res.body.id).toBeDefined();
    expect(res.body.total).toBe(200);
  });
});

	E2E testing with Playwright

> Write E2E tests for the checkout flow
  using Playwright + the staging environment
 
  Critical flows that must be tested:
  1. Happy path: pick a product → checkout → payment → confirmation
  2. Failed payment: wrong card details → error message → retry
  3. Out of stock: product sells out mid-checkout → error + redirect
  4. Session timeout: idle too long → session expires → redirect to login
 
# Example Playwright test
test("complete checkout flow", async ({ page }) => {
  await page.goto("/products");
 
  // pick a product
  await page.click("[data-testid=product-1]");
  await page.click("[data-testid=add-to-cart]");
 
  // go to checkout
  await page.click("[data-testid=cart-icon]");
  await page.click("[data-testid=checkout-btn]");
 
  // fill in payment
  await page.fill("[data-testid=card-number]", "4242424242424242");
  await page.fill("[data-testid=card-expiry]", "12/25");
  await page.fill("[data-testid=card-cvc]", "123");
  await page.click("[data-testid=pay-btn]");
 
  // verify confirmation
  await expect(page.locator("[data-testid=order-confirmed]"))
    .toBeVisible({ timeout: 10000 });
});

Stage 4 — Security checklist

Security is what vibe coders forget most often. Run this checklist every time before you deploy:

# Have Claude run a security audit
> Run a security audit on the code just written
  Check the relevant items from the OWASP Top 10:
 
  □ Injection:
    - SQL: are all queries parameterised?
    - NoSQL: are MongoDB operators sanitised?
    - Command: does exec() or any shell command take input directly?
 
  □ Authentication:
    - Does every sensitive route have an auth check?
    - Is JWT validation correct (verify signature, expiry)?
    - Is there rate limiting on the login endpoint?
 
  □ Authorization (IDOR):
    - Can a user only view and edit their own data?
    - Is ownership checked before data is returned?
 
  □ Sensitive Data:
    - Are passwords hashed with bcrypt (cost >= 10)?
    - Are sensitive fields (password, token) kept out of responses?
    - Are logs free of PII and credentials?
 
  □ Input Validation:
    - Is every input validated and sanitised?
    - File upload: is type checked, size limited, content scanned?
    - URL parameters: is the format validated?
 
  □ Error Handling:
    - Do error messages avoid leaking internal information?
    - Are stack traces kept from reaching the client?
 
  For each item: ✅ pass / ❌ problem found (explain)

Stage 5 — Performance testing and benchmarking

Before deploying you need to know how much load the code can take, and where the bottleneck sits.

	Load testing with k6

# Have Claude write a k6 load test script
> Write a k6 load test for the critical API
 
  Target: /api/products/search
  Expected: p95 latency < 500ms at 100 concurrent users
 
// the k6 script Claude will produce
import http from "k6/http";
import { check, sleep } from "k6";
 
export const options = {
  stages: [
    { duration: "30s", target: 10 },   // warm up
    { duration: "1m",  target: 100 },  // ramp to target
    { duration: "3m",  target: 100 },  // steady state
    { duration: "30s", target: 0 },    // ramp down
  ],
  thresholds: {
    http_req_duration: ["p(95)<500"],  // 95% must be < 500ms
    http_req_failed: ["rate<0.01"],    // error rate < 1%
  },
};
 
export default function() {
  const params = {
    headers: { Authorization: `Bearer ${__ENV.TEST_TOKEN}` },
  };
 
  const searchTerms = ["shirt", "pants", "shoes", "hat"];
  const term = searchTerms[Math.floor(Math.random() * searchTerms.length)];
 
  const res = http.get(`${__ENV.BASE_URL}/api/products/search?q=${term}`, params);
 
  check(res, {
    "status 200": (r) => r.status === 200,
    "has products": (r) => JSON.parse(r.body).products.length >= 0,
    "response time OK": (r) => r.timings.duration < 500,
  });
 
  sleep(1);
}
 
# run with: k6 run -e BASE_URL=https://staging.myapp.com \
#          -e TEST_TOKEN=xxx script.js

	Database query performance with EXPLAIN ANALYZE

> Analyse the performance of this query:
  [paste the Prisma query]
 
  Produce the raw SQL EXPLAIN ANALYZE version
  so we can read the execution plan:
  1. Seq Scan or Index Scan?
  2. Is there a large gap between estimated and actual rows?
  3. Which step burns the most memory?
 
  Then propose:
  - Indexes worth adding
  - A query rewrite if necessary
  - A caching strategy if appropriate
 
# Example EXPLAIN ANALYZE output for Claude to analyse
EXPLAIN ANALYZE
SELECT u.*, COUNT(o.id) as order_count
FROM users u
LEFT JOIN orders o ON o.user_id = u.id
WHERE u.created_at > NOW() - INTERVAL "30 days"
GROUP BY u.id;
 
# Claude will spot the Seq Scan on users and propose:
# CREATE INDEX idx_users_created_at ON users(created_at DESC);

Stage 6 — Observability: logging, metrics, tracing

Good code has to be observable. When something breaks in production you need to find the cause in minutes, not hours.

	Structured logging

> Add structured logging in @src/lib/logger.ts
  Requirements:
  - Use pino (10x faster than winston)
  - Log format: JSON (so it can be parsed)
  - Log levels: error, warn, info, debug
  - Every log must carry: timestamp, level, service,
    requestId, userId (when available), message, data
  - Never log: passwords, tokens, credit cards
  - Production: info level and above only
 
# Example of a good logger call
logger.info({
  event: "payment.processed",
  requestId: ctx.requestId,
  userId: user.id,
  orderId: order.id,
  amount: order.total,
  duration: timer.elapsed(),
  // no card number!
}, "Payment processed successfully");
 
> Add request ID middleware that injects requestId
  into every request context so calls can be traced across services

	Health check endpoint

> Build a /api/health endpoint for monitoring
  It must check:
  □ Database: does it respond to a ping? Does a simple query run?
  □ Redis (if used): does it respond to a ping?
  □ External APIs: are Stripe and SendGrid reachable?
  □ Disk space: more than 20% free?
  □ Memory: heap usage under 80%?
 
  Response format:
  {
    "status": "healthy" | "degraded" | "unhealthy",
    "timestamp": "2025-01-15T10:00:00Z",
    "version": "1.4.2",
    "checks": {
      "database": { "status": "ok", "latency": 5 },
      "redis":    { "status": "ok", "latency": 1 },
      "stripe":   { "status": "ok", "latency": 120 }
    }
  }
 
  - 200: healthy or degraded (some services slow but still working)
  - 503: unhealthy (a core service has failed)
  - Timeout per check: 2 seconds

Stage 7 — Deployment strategy

A good deployment needs a clear rollback plan and as little downtime as possible.

	Zero-downtime deployment checklist

> Build a deployment checklist for [feature]
  with Claude filling in:
 
  ## Pre-deployment
  □ All tests pass (unit, integration, e2e)
  □ Database migration reviewed (backward compatible?)
  □ New environment variables set in production?
  □ Feature flags configured?
  □ Monitoring alerts set up?
 
  ## Migration Strategy
  (important: a migration must stay backward compatible for at least 1 version)
  Step 1: Deploy the migration (add the new column, nullable)
  Step 2: Deploy the app (start using the new column)
  Step 3: Backfill the old data (background job)
  Step 4: Make the column NOT NULL (next release)
 
  ## Deployment Steps
  1. Deploy to staging → smoke test
  2. Deploy to production (canary: 5% of traffic)
  3. Monitor for 15 minutes (error rate, latency)
  4. Expand to 50% → monitor
  5. Full rollout → monitor for 1 hour
 
  ## Rollback Plan
  Trigger: error rate > 1% or p95 > 2s
  Time to rollback: < 5 minutes
  Command: [the actual rollback command]

QA testing — checking before production

QA (quality assurance) is the process of verifying that a feature behaves correctly in every situation. Claude can help you design and run QA systematically.

A QA test plan with Claude

> Build a QA test plan for [Feature Name]
 
  Feature details:
  [describe the feature you built]
 
  Produce a test plan covering:
 
  ## Functional Testing
  - Happy path scenarios (every user journey)
  - Negative scenarios (bad input, insufficient permissions)
  - Boundary testing (min/max values)
 
  ## Browser/Device Compatibility
  - Chrome, Firefox, Safari (latest)
  - Mobile: iOS Safari, Android Chrome
  - Responsive: 320px, 768px, 1280px, 1920px
 
  ## Accessibility Testing
  - Is keyboard navigation complete?
  - Is it screen reader compatible?
  - Colour contrast to WCAG 2.1 AA?
  - Are focus indicators visible?
 
  ## Data Integrity
  - Is the data saved correctly?
  - Does it display correctly after a refresh?
  - Concurrent users: any race condition?
 
  ## Format: Test Case Table
  | ID | Scenario | Steps | Expected | Priority |

An automated QA checklist for vibe coding

Run this checklist every time before merging a PR:

# .claude/commands/qa-check.md
# QA Pre-merge Checklist
 
# Run every automated check and report the results
 
1. Run: npm run typecheck
   ✓ no TypeScript errors allowed
 
2. Run: npm run lint
   ✓ no ESLint errors (warnings are OK)
 
3. Run: npm test
   ✓ must pass 100% (0 failures)
   ✓ coverage must be >= 70% for new code
 
4. Run: npm run build
   ✓ the build must succeed with no errors
 
5. Run the security check:
   npm audit --audit-level=high
   ✓ no high/critical vulnerabilities
 
6. Inspect git diff --stat
   ✓ no .env committed
   ✓ no stray console.log
   ✓ no TODO that should not be in prod
 
7. Manual testing on localhost:
   - The most important happy path
   - The error case most likely to occur
 
Report the result as: ✅ pass / ❌ fail (with details)

Visual regression testing

Checks that the UI has not changed unintentionally after a refactor or a dependency upgrade:

# install Playwright screenshot testing
npm install @playwright/test
 
> Write visual regression tests for the important pages:
  - Homepage
  - Product listing
  - Checkout flow (3 steps)
  - User profile
 
// visual.test.ts
import { test, expect } from "@playwright/test";
 
test("homepage visual", async ({ page }) => {
  await page.goto("/");
  await page.waitForLoadState("networkidle");
  await expect(page).toHaveScreenshot("homepage.png", {
    maxDiffPixels: 100,  // allow up to 100 differing pixels
  });
});
 
# first time: run to create the baseline
npx playwright test visual --update-snapshots
 
# afterwards: compare against the baseline
npx playwright test visual
# if the UI changed → the test fails → approve it or fix it

A/B testing and feature flags

A/B testing means running two versions of a feature side by side with real users to measure which is better. Feature flags control who sees which version.

Feature flags — controlling the release

Feature flags let you deploy the code first and enable it for users later, which lowers the risk of any single deployment.

# Have Claude implement the feature flag system
> Build a feature flag system for the project
  Requirements:
  - Store flags in the database (changeable without a redeploy)
  - Target: all users / named users / a percentage
  - Admin UI: turn a flag on or off immediately
  - Cache: 1 minute (do not hit the DB on every request)
 
  Schema:
  model FeatureFlag {
    id          String   @id @default(cuid())
    key         String   @unique  // "new-checkout"
    enabled     Boolean  @default(false)
    rolloutPct  Int      @default(0)   // 0-100%
    userIds     String[] // specific users
    description String
    createdAt   DateTime @default(now())
    updatedAt   DateTime @updatedAt
  }
 
  Usage in code:
  if (await featureFlag.isEnabled("new-checkout", user.id)) {
    // show new checkout
  } else {
    // show old checkout
  }

An A/B test framework

An A/B test measures which variant produces the better conversion rate.

> Build an A/B test framework that:
  1. Assigns a user to a variant consistently
     (the same user always gets the same variant)
  2. Tracks the events that matter (the conversion goal)
  3. Calculates statistical significance
  4. Shows the results on a dashboard
 
  Example A/B test: CTA button
  Variant A (control): "Add to cart" (blue)
  Variant B (treatment): "Buy now" (red)
  Goal: click rate
  Duration: 2 weeks, or n=1000 per variant
 
// ab-test.ts
export async function getVariant(
  testId: string,
  userId: string
): Promise<"control" | "treatment"> {
  // deterministic hash so the assignment stays consistent
  const hash = crypto
    .createHash("sha256")
    .update(`${testId}:${userId}`)
    .digest("hex");
  const num = parseInt(hash.substring(0, 8), 16);
  return num % 2 === 0 ? "control" : "treatment";
}
 
export async function trackEvent(
  testId: string,
  userId: string,
  event: "impression" | "conversion"
) {
  const variant = await getVariant(testId, userId);
  await db.abTestEvent.create({
    data: { testId, userId, variant, event }
  });
}

Analysing A/B test results with Claude

# after two weeks of collecting data
> Analyse the results of the "cta-button-color" A/B test
  Data:
  Control:   1,024 impressions, 87 conversions (8.5%)
  Treatment: 1,031 impressions, 112 conversions (10.9%)
 
  Please:
  1. Calculate statistical significance (p-value)
     using a Chi-square test or Fisher exact test
  2. Calculate the confidence interval for the difference
  3. Tell me whether the result is significant (p < 0.05)
  4. If significant: what do you recommend doing?
  5. If not significant: how large a sample would we need?

Benchmarking and profiling

Benchmarking means measuring performance systematically; profiling means finding which part of the code is slow. Both are essential skills for making production code fast and resource-efficient.

Micro-benchmarks with Vitest

> Write a benchmark comparing algorithm A with algorithm B:
 
  Algorithm A: Array.filter().map()
  Algorithm B: a single Array.reduce()
 
  Test scenarios:
  - Arrays of 100, 1,000, 10,000, 100,000 elements
  - Run each scenario 1000 times
  - Report: mean, median, p99, memory usage
 
// benchmark.test.ts
import { bench, describe } from "vitest";
 
describe("search algorithms", () => {
  const data = Array.from({ length: 10000 },
    (_, i) => ({ id: i, value: Math.random() })
  );
 
  bench("filter + map", () => {
    data.filter(x => x.value > 0.5).map(x => x.id);
  });
 
  bench("reduce", () => {
    data.reduce((acc, x) => {
      if (x.value > 0.5) acc.push(x.id);
      return acc;
    }, [] as number[]);
  });
});
 
# run with: npx vitest bench

Node.js profiling with —prof

> Help me set up profiling for a Node.js app
  to find the CPU bottleneck
 
  # run the app with the profiler
  node --prof dist/server.js
 
  # send traffic (load test)
  k6 run profile-test.js
 
  # stop the app, then process the profile
  node --prof-process isolate-*.log > profile.txt
 
  # then have Claude analyse it
> Here is a Node.js CPU profile:
  [paste the top functions from profile.txt]
 
  Analyse:
  1. Which function burns the most CPU?
  2. Is it a real bottleneck or a false positive?
  3. What is the right way to optimise it?
 
  # Alternative: use clinic.js (easier)
  npm install -g clinic
  clinic doctor -- node dist/server.js
  # opens the browser automatically and shows a flame graph

Database query profiling

> Build a database profiling setup:
  1. Turn on query logging in Prisma:
  const prisma = new PrismaClient({
    log: [
      { level: "query", emit: "event" },
      { level: "error", emit: "stdout" },
    ],
  });
 
  prisma.$on("query", (e) => {
    if (e.duration > 100) {  // log queries > 100ms
      logger.warn({
        query: e.query,
        params: e.params,
        duration: e.duration,
      }, "Slow query detected");
    }
  });
 
  2. Add pg_stat_statements in PostgreSQL
     to track query statistics:
  CREATE EXTENSION pg_stat_statements;
 
  -- view the slow queries
  SELECT query, calls, mean_exec_time, total_exec_time
  FROM pg_stat_statements
  WHERE mean_exec_time > 100
  ORDER BY total_exec_time DESC
  LIMIT 10;

Error handling and recovery strategy

A production system has to handle failure gracefully. The goal is not merely “no errors” — it is knowing what to do when they happen.

Error taxonomy — classifying errors correctly

Error class	Example	How to handle	Retry?
Transient	Network timeout, DB connection blip	Retry with backoff	Yes
Validation	Bad email format, missing field	Return 400 with details	No
Business rule	Expired coupon, product out of stock	Return 422 with a user message	No
Auth	Expired token, insufficient rights	Return 401/403	Sometimes
Dependency	Stripe down, SendGrid failing	Fall back or queue	Yes
System	OOM, disk full, a bug in the code	Alert the team, log in detail	Sometimes

The retry pattern with exponential backoff

> Build a retry utility for use across the project
 
// retry.ts
export async function withRetry<T>(
  fn: () => Promise<T>,
  options: {
    maxAttempts?: number;   // default: 3
    initialDelay?: number;  // default: 1000ms
    maxDelay?: number;      // default: 30000ms
    backoffFactor?: number; // default: 2 (exponential)
    retryIf?: (error: Error) => boolean;
    onRetry?: (attempt: number, error: Error) => void;
  } = {}
): Promise<T> {
  const {
    maxAttempts = 3,
    initialDelay = 1000,
    maxDelay = 30000,
    backoffFactor = 2,
    retryIf = () => true,
    onRetry = () => {},
  } = options;
 
  let lastError: Error;
 
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (error) {
      lastError = error as Error;
 
      if (attempt === maxAttempts || !retryIf(lastError)) {
        throw lastError;
      }
 
      const delay = Math.min(
        initialDelay * Math.pow(backoffFactor, attempt - 1),
        maxDelay
      );
      const jitter = delay * 0.1 * Math.random(); // ±10% jitter
 
      onRetry(attempt, lastError);
      await new Promise(resolve => setTimeout(resolve, delay + jitter));
    }
  }
 
  throw lastError!;
}
 
// Usage
const result = await withRetry(
  () => stripe.paymentIntents.create(params),
  {
    maxAttempts: 3,
    retryIf: (err) => err.message.includes("network"),
    onRetry: (attempt, err) => logger.warn({ attempt, err }, "Retrying payment"),
  }
);

The circuit breaker pattern

Stops the system from repeatedly calling an external service that is already failing. It lets you fail fast and recover sooner.

> Implement a circuit breaker for external API calls
  States:
  - CLOSED: normal operation
  - OPEN: stop sending requests (fail fast)
  - HALF_OPEN: try one request; if it succeeds → CLOSED
 
  Config:
  - failureThreshold: 5 failures → OPEN
  - successThreshold: 2 successes → CLOSED (from HALF_OPEN)
  - timeout: 30 seconds before HALF_OPEN
 
// using the opossum library
npm install opossum
 
import CircuitBreaker from "opossum";
 
const stripeBreaker = new CircuitBreaker(
  (params) => stripe.paymentIntents.create(params),
  {
    timeout: 5000,          // 5s timeout per request
    errorThresholdPercentage: 50,  // 50% errors → OPEN
    resetTimeout: 30000,    // 30s before HALF_OPEN
    volumeThreshold: 10,    // needs 10+ requests before it can open
  }
);
 
stripeBreaker.on("open",     () => logger.error("Circuit OPEN: Stripe"));
stripeBreaker.on("halfOpen", () => logger.warn("Circuit HALF_OPEN: Testing Stripe"));
stripeBreaker.on("close",    () => logger.info("Circuit CLOSED: Stripe recovered"));
 
// Usage (transparent to caller)
const intent = await stripeBreaker.fire(params);

Graceful shutdown

When a server has to restart, the requests already in flight must finish first rather than being dropped halfway.

> Implement graceful shutdown for an Express server
  Requirements:
  - Catch the SIGTERM signal
  - Stop accepting new requests
  - Wait for in-flight requests to finish (timeout: 30s)
  - Close the database connection
  - Exit with code 0
 
// shutdown.ts
let isShuttingDown = false;
 
export function gracefulShutdown(server: Server) {
  process.on("SIGTERM", async () => {
    logger.info("SIGTERM received, starting graceful shutdown");
    isShuttingDown = true;
 
    server.close(async () => {
      try {
        await prisma.$disconnect();
        await redis.quit();
        logger.info("Graceful shutdown complete");
        process.exit(0);
      } catch (err) {
        logger.error(err, "Error during shutdown");
        process.exit(1);
      }
    });
 
    // force shutdown after 30s
    setTimeout(() => {
      logger.error("Forced shutdown after timeout");
      process.exit(1);
    }, 30000);
  });
}
 
// middleware: reject new requests during shutdown
app.use((req, res, next) => {
  if (isShuttingDown) {
    res.set("Connection", "close");
    return res.status(503).json({ error: "Server shutting down" });
  }
  next();
});

Production incident management

When something goes wrong in production, how you respond has a large effect on the resulting downtime. Claude helps with both the debugging and the communication.

Incident response playbook

1	Detect — know that something is wrong An alert from the monitoring you set up, or a wave of user reports. Claude Code can query the logs immediately

# check the state quickly
> Query the logs for the last 15 minutes:
  Run: grep "ERROR" /var/log/app/app.log | tail -50
  Or query Cloudwatch:
  aws logs filter-log-events --log-group-name /app/production
    --start-time $(date -d "15 minutes ago" +%s000)
    --filter-pattern "ERROR"
 
  Then summarise:
  1. Which class of error occurs most?
  2. When did it start?
  3. Which endpoints are affected?

2	Triage — assess the severity Give Claude what you have; it helps assess impact and priority

> Incident assessment:
  What we know:
  - Error rate: [X%] (normally < 0.1%)
  - Affected users: [N]
  - Affected features: [list]
  - Duration: [X minutes]
 
  Assess:
  1. Severity: P1 (system down) / P2 (a key feature broken) / P3 (minor impact)
  2. Who needs to be told: team lead, CTO, users?
  3. Timeline for a fix: hours or days
  4. Any quick mitigation we can apply right now?

3	Mitigate — reduce the impact first Apply a workaround to cut downtime, then go looking for the root cause

# mitigations you can apply fast
> We have found that the payment feature is broken
  and we do not yet know why
  Propose mitigation options we can apply within 5 minutes:
  - Feature flag: turn payments off temporarily
  - Maintenance mode: redirect to a notice page
  - Rollback: return to the previous version
  - Scale up: add servers if it is a load issue
  Give the actual command for each option

4	Root cause analysis Once the impact is contained, find the root cause systematically

5	Post-mortem After the fix, write a post-mortem so it does not happen again

> Write a post-mortem document for this incident:
  Details:
  - Started: [time]
  - Resolved: [time]
  - Duration: [X minutes]
  - Impact: [N users, N transactions affected]
 
  Template:
  ## Timeline
  [time] what happened
 
  ## Root Cause
  Describe the real root cause (not the symptom)
 
  ## Contributing Factors
  What made the problem worse or slowed the fix
 
  ## What Went Well
  What the response got right
 
  ## Action Items
  | Action | Owner | Due Date | Priority |
  |--------|-------|----------|----------|
  | Add a test that catches this bug | @john | 2025-01-20 | P1 |
  | Add an alert for this error pattern | @jane | 2025-01-18 | P1 |
 
  ## Prevention
  How will we stop this from recurring?

Summary: the production-ready checklist

Everything above, collected into one checklist to run before a real production deploy:

# .claude/commands/prod-ready.md
# Production Readiness Checklist
 
## Code Quality
□ TypeScript strict mode: no any
□ Tests: unit + integration + e2e all pass
□ Test coverage: >= 70% for new code
□ Linting: no errors
□ Build: succeeds with no significant warnings
 
## Security
□ Authentication: every protected route has an auth check
□ Authorization: ownership verified before data is returned
□ Input validation: every input goes through validation
□ SQL: no raw string concatenation
□ Secrets: no hardcoded credentials
□ Dependencies: npm audit reports no high/critical
 
## Performance
□ Database: indexes cover the frequently used queries
□ N+1: no N+1 query in a critical path
□ Load test: meets the performance target
□ Caching: expensive operations are cached
 
## Observability
□ Logging: structured logs in place
□ Error tracking: Sentry configured
□ Health check: /api/health works
□ Alerts: monitoring alerts set up
 
## Reliability
□ Error handling: every async call has try/catch
□ Retry logic: transient errors are retried
□ Graceful shutdown: SIGTERM handled
□ Circuit breaker: external APIs are protected
 
## Deployment
□ Migration: backward compatible
□ Rollback: plan is clear and tested
□ Feature flags: configured
□ Runbook: updated
 
□ All items: ✅ ready to deploy | ❌ must be fixed first

Glossary for this chapter

5 Whys	A technique for finding the root cause by asking “why” repeatedly until you reach the real origin
Root cause	The actual cause of a problem, not the symptom you can see
Contract-first	Defining the interface/API contract before implementing, to reduce ambiguity
Circuit breaker	A pattern that stops sending requests to a failing service so you fail fast
Exponential backoff	Retrying with a longer wait each time, so a recovering service is not hammered
Graceful shutdown	A shutdown that waits for in-flight requests to finish rather than dropping them
Feature flag	A switch that turns a feature on or off without a redeploy
A/B test	Running 2 variants side by side with real users to measure which is better
Statistical significance	The statistical confidence that a difference is not down to chance
p-value	A number saying how unlikely the observed result is under chance alone (< 0.05 = significant)
Benchmark	Measuring performance systematically for comparison
Profiling	Finding which part of the code consumes the most CPU/memory
Flame graph	A graph of the call stack by CPU usage, used to find bottlenecks
EXPLAIN ANALYZE	The SQL command that shows a query’s real execution plan
Load test	Testing a system under heavy load to find its breaking point
Canary deployment	Deploying to a small % of users first, then expanding
Post-mortem	The analysis after an incident, to find the cause and prevent a repeat
MTTR	Mean Time To Recovery — the average time taken to resolve an incident
Error budget	The amount of error/downtime an SLA tolerates (e.g. 0.1% per month)