A/B Tests, QA, and Measuring User Harm

6.1 The testing layers

Agents write tests very quickly, which is dangerous: they will produce a great many tests that verify what the code already does, instead of what the spec requires. Define the layers clearly, so each one has a kind of bug it is responsible for catching.

Layer	Catches	Share and time	How far an agent can go
Unit	Logic errors inside one function	~70% / under 10s	Very well, provided they are written from the spec, not from the code
Property	Cases nobody thought of — invariants that must always hold	~10% / under 30s	Very well, provided a human states what the invariants are
Contract	Accidental changes to an API’s shape	~5% / under 60s	Very well; can be generated from OpenAPI
Integration	Assembly with the DB, queues, external services	~10% / 1–5 min	Needs fixtures a human set up first
E2E	Real user journeys breaking	~5% / 5–20 min	Fragile; a human should choose which journeys are worth maintaining
Golden	Output changing unintentionally	Supplementary	Excellent for data pipelines, prompts, renderers

Property tests — where an agent adds the most value

from decimal import Decimal
from hypothesis import given, strategies as st

money = st.decimals(min_value=0, max_value=1_000_000, places=2)

@given(subtotal=money, pct=st.integers(min_value=0, max_value=100))
def test_percent_discount_invariants(subtotal, pct):
    result = apply_percent_coupon(subtotal, pct)

    # invariant 1: the net total is never negative, whatever the input
    assert result.total >= Decimal("0")

    # invariant 2: the net total never exceeds the original
    assert result.total <= subtotal

    # invariant 3: discount + total always equals the original (no money lost to rounding)
    assert result.discount + result.total == subtotal

    # invariant 4: never more than two decimal places
    assert result.total.as_tuple().exponent >= -2

A human states what those four invariants are — that is business knowledge. The agent finds the inputs that break them, which hypothesis does far better than a person. This is where the human-agent division of labour is clearest.

LLM-as-judge in the gate

Never let an LLM decide pass or fail in a gate that blocks a merge. The result is not stable, CI goes red at random, and the team learns to hit re-run until it is green — which destroys the credibility of the whole gate. Use LLM-as-judge as a supplementary report posted in a PR comment (“three things worth a closer look”), but never give it the power to block.

6.2 A/B tests

In agentic work, “A/B” means two very different things. Keep them apart.

Type 1: A/B between two agent outputs (offline)

Use it when more than one implementation is genuinely reasonable. Have the agent do both in separate worktrees, then measure against criteria fixed in advance.

# Two worktrees, working in parallel
git worktree add ../impl-a -b exp/coupon-strategy-a
git worktree add ../impl-b -b exp/coupon-strategy-b

# Same spec in each, different constraint
(cd ../impl-a && claude -p "implement docs/specs/coupon.md \
  using a strategy pattern, one class per coupon type")
(cd ../impl-b && claude -p "implement docs/specs/coupon.md \
  as a single function with match-case, no new classes")

# Measure against criteria set BEFORE seeing the results — this matters
./scripts/compare.sh ../impl-a ../impl-b

Measurable comparison criteria, fixed before the run and not changed after:

Criterion	Measured with
Does the same test suite pass in both?	If not, discard immediately
Lines added	Fewer is better
Highest cyclomatic complexity per function	radon cc
Number of files touched	Blast radius
p95 latency on the same benchmark	pytest-benchmark
How many places a third feature would need to change	Test it by having the agent actually do it

The last criterion is the most valuable and the one most often skipped. The question is “which design absorbs the next change better”, and it is genuinely testable: have the agent implement the next requirement on both branches and see which diff is smaller.

Type 2: A/B with real users (online)

This tests a business hypothesis, not whether the code is correct. The code must already have passed the whole gate. A/B is not a substitute for QA.

The correct order:
gate green → staging → 5% canary → 50/50 A/B → 100% rollout
                          ▲
              user harm is measured here, not during the A/B

Fix these before opening an A/B:

• Exactly one primary metric, such as conversion rate
• Three to five guardrail metrics that may not get worse
• A minimum sample size computed in advance — no peeking and stopping on a good day
• A minimum duration of at least one full week, to absorb day-of-week effects
• An emergency stop: a guardrail touching its line stops the test immediately, without waiting for the sample

6.3 User harm — measuring whether you have hurt anyone

This is the part most often missing when a team accelerates with AI, because shipping speed rises while damage-detection speed stays where it was. That gap is where users get hurt.

Level	Example	Detected by	Acceptable response time
L1 Annoying	Page 300ms slower, a button shifts	RUM, p95 latency	Fix this sprint
L2 Broken	Cannot place an order, upload fails	Error rate, funnel drop	Fix within 24h
L3 Wrong data	Wrong amounts, over-applied discounts	Reconciliation job	Roll back immediately
L4 Lost data	Overwritten records, a bad migration	Row count drift, backup diff	Prevention only; recovery is hard
L5 Leaked data	Cross-tenant visibility, PII in logs	Tenancy tests, log scanner	Must be blocked at the gate

Where AI adds the most risk

L4 and L5, because these are failures that do not show symptoms immediately. An agent writes a migration with a wrong backfill, or logs a whole user object, and every test stays green. That is why structural tests — a tenancy scan, a PII scan — are necessary, rather than relying on behavioural tests alone.

Guardrail metrics — numbers that may not get worse

Set these in monitoring and wire them to automatic rollback:

Metric	Threshold	Window
error_rate_5xx	< 0.5%	5 minutes
p95_latency	< baseline+20%	5 minutes
checkout_success	> baseline−2%	15 minutes
payment_mismatch	= 0	Every minute (L3 — never acceptable)
cross_tenant_reads	= 0	Every minute (L5 — never acceptable)
row_count_drift	< 0.1%	Against 24 hours earlier (L4)
support_ticket_rate	< baseline+30%	1 hour (the signal humans give you)

The rule: any guardrail touching its line triggers automatic rollback, without waiting for human approval. Investigate afterwards. The cost of an unnecessary rollback is far below the cost of thirty more minutes of damage.

Blast radius — bounding the damage before it happens

Every PR answers four questions in its description, as required fields in the template:

1. If this code is wrong in the worst way, how many users are affected?
2. What level of harm is it (L1–L5)?
3. How many minutes until it is detected, and by what?
4. How is it rolled back, how long does that take, and does it touch data?

If question 2 is L3 or worse and question 3 is “I do not know”, it may not be merged. Go back and add the detection first.

Tools for reducing blast radius, in order of strength:

An instantly disableable feature flag        ← every feature touching money or data
Canary by percentage of users
Canary by tenant (internal tenants first)
Shadow mode (run in parallel, compare, do not use the result)   ← best for L3
Dual-write plus reconciliation                                  ← for migrations

Shadow mode — the safest approach for logic that touches money

async def calculate_total(order: Order) -> Money:
    old = legacy_calculate(order)                 # the existing path — its result is used

    if flags.enabled("coupon_v2_shadow", order.tenant_id):
        try:
            new = await coupon_v2_calculate(order)  # the new path — result discarded
            if new != old:
                metrics.increment("coupon_v2.mismatch")
                log.warning("shadow mismatch",
                            order_id=order.id, old=str(old), new=str(new))
        except Exception:
            metrics.increment("coupon_v2.error")
            log.exception("shadow failed")          # never re-raise
 
    return old                                      # the user always gets the old result

Switch to the new path when mismatches have been zero for seven consecutive days at 100% real traffic.

Where an agent wrote the financial or otherwise critical logic, shadow mode always pays for itself, because it converts “is this code correct?” into real data from real traffic, with nobody harmed along the way.

What this chapter settles

Give each test layer a defined job. Humans state the invariants; agents find the inputs that break them. Never let an LLM block a merge. Keep offline and online A/B separate. And measure user harm with guardrails wired to automatic rollback — because AI raises your shipping speed without raising your detection speed.

The next chapter walks a real feature through nine stages from spec to rollout, naming which stages a human must do personally.