Loop, Gate, and Red / Green

The gate is the heart of the whole system. Make it weak and the agent learns to turn it green without making the program correct. Make it too strict or too slow and the loop never closes, and people stop using it.

5.1 What makes a good gate

Property	What it means in practice
Deterministic	Ten runs give the same result. No test depends on time, network, or ordering
Fast enough	The quick set under 60 seconds, the full set under 10 minutes. Slower than that and the loop breaks
Loud	The error says what is wrong, where, what was expected and what was received — not merely exit 1
Layered	Split into stages that stop on the first failure rather than waiting for later ones
Tamper-evident	The gate files are outside what the agent may edit, and CI runs the same set as local

5.2 verify.sh — the definition of “passing”

#!/usr/bin/env bash
# THE GATE — the single source of truth for "the work is done"
# The agent may not edit this file (enforced by a PreToolUse hook and CODEOWNERS)
set -uo pipefail

FAILED=0
STAGE_LOG=$(mktemp)

run_stage () {
  local name="$1"; shift
  printf "\n\033[1m▸ %s\033[0m\n" "$name"
  local start=$SECONDS
  if "$@" 2>&1 | tee -a "$STAGE_LOG"; then
    printf "  \033[32mPASS\033[0m (%ss)\n" "$((SECONDS-start))"
  else
    printf "  \033[31mFAIL\033[0m (%ss)\n" "$((SECONDS-start))"
    FAILED=1
    [ "${FAIL_FAST:-1}" = "1" ] && finish
  fi
}

finish () {
  echo
  if [ $FAILED -eq 0 ]; then
    echo "════ ALL GATES PASSED ════"; exit 0
  else
    echo "════ GATE FAILED ════"; exit 1
  fi
}

# ── STAGE 1: shape (fastest, catch it first) ─────────────
run_stage "format"      uv run ruff format --check .
run_stage "lint"        uv run ruff check .
run_stage "types"       uv run mypy src --strict

# ── STAGE 2: logical correctness ─────────────────────────
run_stage "unit"        uv run pytest tests/unit -q --timeout=30
run_stage "property"    uv run pytest tests/property -q --hypothesis-seed=0

# ── STAGE 3: assembly ────────────────────────────────────
run_stage "integration" uv run pytest tests/integration -q --timeout=120
run_stage "contract"    uv run schemathesis run openapi.json --checks all

# ── STAGE 4: structural quality ──────────────────────────
run_stage "coverage"    uv run pytest --cov=src --cov-fail-under=80 -q
run_stage "arch-rules"  uv run pytest tests/architecture -q     # dependency rules
run_stage "security"    uv run bandit -r src -q -ll
run_stage "deps-audit"  uv run pip-audit --strict
run_stage "migrations"  ./scripts/check_migrations.sh

# ── STAGE 5: anti-cheating ───────────────────────────────
run_stage "no-cheating" ./scripts/check_no_cheating.sh

finish

Checking that nothing was made green by silencing the tools:

#!/usr/bin/env bash
set -uo pipefail

BASE="${BASE_REF:-origin/main}"
DIFF=$(git diff "$BASE"...HEAD -- 'src/**' 'tests/**')
BAD=0

check () {
  local pattern="$1" msg="$2"
  # Only look at lines that were ADDED in the diff
  if echo "$DIFF" | grep -E "^\+" | grep -Eq "$pattern"; then
    echo "  ✗ $msg"; BAD=1
  fi
}

check '@pytest\.mark\.(skip|xfail)' "skip/xfail added to a test"
check '#\s*type:\s*ignore'          "type: ignore added"
check '#\s*noqa'                    "noqa added"
check 'assert True'                 "an assertion that tests nothing"
check 'except\s*:\s*pass'           "an exception silently swallowed"

# The test count must not go down
OLD=$(git grep -c "^def test_" "$BASE" -- tests | awk -F: '{s+=$3} END {print s+0}')
NEW=$(git grep -c "^def test_" HEAD -- tests | awk -F: '{s+=$3} END {print s+0}')
if [ "$NEW" -lt "$OLD" ]; then
  echo "  ✗ test count fell from $OLD to $NEW"; BAD=1
fi

[ $BAD -eq 0 ] && echo "  no signs of gate evasion"
exit $BAD

Why a “no-cheating” stage is necessary

Put an agent under the condition “make the tests pass” and it will find the shortest route — and sometimes the shortest route is making the tests disappear. This is not model dishonesty; it is what happens when a goal is measured by a proxy. The remedy is to make the proxy hard to game, not to ask nicely for it not to be gamed.

5.3 Red / Green that an agent can genuinely do

TDD works better with an agent than with a human in one respect: a red test is machine-readable feedback, which is exactly what an agent wants most. But the order has to be enforced, or the agent writes the implementation first and then tests that confirm what it just wrote — which is worthless.

[RED-1]      Write tests from the spec only. Do not read or write the implementation
[RED-2]      Run them → they must be red, and red for the right reason
             (AssertionError = good | ImportError/NameError = not yet a usable red)
[RED-3]      Show the red output, with the number of failing tests
[GREEN]      Write the smallest implementation that turns them green.
             Add no feature the tests did not ask for
[REFACTOR]   Restructure with the tests green throughout, re-running after each change

A prompt for driving red-green:

Task: implement docs/specs/coupon.md

Follow this order strictly, and report every step with real output:

Step 1 (RED)
- Read docs/specs/coupon.md only. Do not open src/domain/coupon.py
- Write tests in tests/unit/test_coupon.py covering every rule in the spec,
  including every case the spec says must be rejected
- Run `uv run pytest tests/unit/test_coupon.py -q` and paste the output
- Confirm each failure is an AssertionError or missing business logic.
  If it fails with ImportError, create a stub raising NotImplementedError and re-run
- Stop here and wait for me to confirm the tests are sufficient

Step 2 (GREEN) — begin only when I say "go"
- Write src/domain/coupon.py until the tests are green
- Do not edit the test file in this step, not one line
- If you believe a test is wrong, stop and explain. Do not fix it yourself
- Run verify-fast.sh after every change

Step 3 (REFACTOR)
- Improve readability, running the tests after every change
- Report a summary diff of what was restructured and why

The technique: stopping between RED and GREEN

The highest-value moment for a human is between RED and GREEN. Reading the tests the agent wrote takes three to five minutes and determines the quality of the entire piece of work, because the implementation will always run toward the tests. If the tests are wrong, the code will be wrong with them, and no gate will catch it.

5.4 Controlling the loop: ceilings and stall detection

#!/usr/bin/env bash
# Headless loop driver — works with claude -p, codex exec, or cursor-agent
set -uo pipefail

TASK="${1:?usage: ./agent-loop.sh 'task description'}"
MAX_ITER="${MAX_ITER:-6}"
STALL_LIMIT=2          # stop if the failing-test count does not fall for N rounds

prev_fail=999999
stall=0

for i in $(seq 1 "$MAX_ITER"); do
  echo "═══ round $i / $MAX_ITER ═══"

  # --- 1. one round of agent work ---
  claude -p "$TASK
Context: read AGENTS.md, memory/pitfalls.md, and the previous round's output in /tmp/gate.log
Take one step that brings verify closer to green, then stop. Do not try to fix everything at once." \
    --allowedTools "Read,Edit,Write,Bash(uv run pytest:*),Bash(uv run ruff:*)" \
    --output-format stream-json > "/tmp/agent_$i.log"

  # --- 2. run the gate ---
  ./scripts/verify-fast.sh > /tmp/gate.log 2>&1
  if [ $? -eq 0 ]; then
    echo "GREEN at round $i"
    ./scripts/verify.sh                  # full set, to confirm
    exit $?
  fi

  # --- 3. is it making progress? ---
  cur_fail=$(grep -oE '[0-9]+ failed' /tmp/gate.log | head -1 | grep -oE '[0-9]+' || echo 999999)
  echo "  still failing: $cur_fail (previous round: $prev_fail)"

  if [ "$cur_fail" -ge "$prev_fail" ]; then
    stall=$((stall+1))
    if [ "$stall" -ge "$STALL_LIMIT" ]; then
      echo "stopping: no progress for $STALL_LIMIT rounds — the spec is probably unclear, or the task too large"
      git stash push -m "agent-stalled-$(date +%s)"   # keep it for a human, do not discard
      exit 3
    fi
  else
    stall=0
  fi
  prev_fail=$cur_fail

  # --- 4. safety: the code should not balloon ---
  ADDED=$(git diff --numstat | awk '{s+=$1} END {print s+0}')
  if [ "$ADDED" -gt 800 ]; then
    echo "stopping: $ADDED lines added, more than one PR should contain"
    exit 4
  fi
done

echo "stopping: $MAX_ITER rounds and still not green"
echo "A human should read /tmp/gate.log and decide whether to split the task or fix the spec"
exit 2

Every stop condition worth having

Condition	Signal	What to do next
Success	The whole gate is green	Open the PR
Ceiling reached	N rounds, still red	A human reads the log and splits the task
No progress	Failure count flat for two rounds	The spec is probably ambiguous — go back and fix it
Ballooning code	Diff over 800 lines	Too large for one loop; break it up
Touching forbidden things	The hook blocked three or more times	The agent has the boundaries wrong — fix AGENTS.md
Going in circles	The diff repeats a previous round	Stuck in a local minimum; reset and change approach
Budget exhausted	Tokens or time over the limit	Stop and report cost against result

A rule from experience

If the agent cannot get it passing within three rounds, the problem is usually not the agent — it is an ambiguous spec or a task that is too large. Raising the ceiling to 15 rounds almost never helps; it only makes the code messier. The correct fix is to go back and fix the spec, not to extend the ceiling.

What this chapter settles

The gate defines “done”, so it must be deterministic, fast enough, articulate, layered, and beyond the agent’s reach. It needs a stage dedicated to catching evasion, because a goal measured by a proxy will always attract shortcuts. Enforce red before green, always. And when a loop stops making progress, stop it rather than raising its ceiling.

The next chapter covers the layers of testing, both meanings of A/B, and how to measure whether you have hurt your users yet.