Measuring What a Model Actually Consumes

One question can be answered before you download anything: will it fit? The other two can only be answered after it runs: is it fast enough, and where is the bottleneck?

5.1 What VRAM is made of

actual VRAM = model weights + KV cache + activations + runtime overhead

Component	Character
Model weights	Fixed, and computable in advance with the formula from chapter 2
KV cache	Grows with context length and with concurrent users — the part people forget
Activations	Intermediate values during computation; depends on batch size
Overhead	CUDA context and runtime buffers, roughly 0.5–2 GB

The KV cache formula

KV cache (bytes) = 2 × layers × kv_heads × head_dim × ctx_len × batch × bytes_per_value

Term	Meaning
2	Because both Key and Value are stored
layers	The model’s layer count, from config.json
kv_heads	KV head count. Modern models use GQA, which makes this much smaller than the attention head count
head_dim	hidden_size ÷ num_attention_heads
bytes	2 for FP16, 1 for FP8

Worked through on a typical 8B model with 36 layers, 8 kv_heads and head_dim 128:

Condition	Working	Result	Note
8K context, 1 user, FP16	2 × 36 × 8 × 128 × 8192 × 1 × 2	1.2 GB
32K context, 1 user, FP16	2 × 36 × 8 × 128 × 32768 × 1 × 2	4.8 GB	Nearly as large as the Q4 model itself
32K context, 8 concurrent, FP16	4.8 × 8	38.6 GB	Where real systems break
32K context, 8 concurrent, FP8	38.6 ÷ 2	19.3 GB	Why --kv-cache-dtype fp8 is worth turning on

The most expensive lesson a beginner learns

Everything works while you test alone on your own machine, then the system falls over the day ten real people use it. The KV cache grows linearly with concurrent users, multiplied by context length. Always compute VRAM at your expected peak user count, never at one.

5.2 Tools for watching resource use

# Whole-card overview, refreshed every second
nvidia-smi -l 1

# Only the numbers you care about, in a form you can graph
nvidia-smi --query-gpu=timestamp,name,memory.used,memory.total,\
utilization.gpu,utilization.memory,temperature.gpu,power.draw,clocks_throttle_reasons.active \
  --format=csv -l 2

# Log to a file while a benchmark runs
nvidia-smi --query-gpu=timestamp,memory.used,utilization.gpu,power.draw \
  --format=csv -l 1 > gpu_log.csv &

# An interactive display (install with apt install nvtop)
nvtop

# Which process is holding VRAM
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv

# CPU / RAM / disk
htop ; free -h ; df -h

Measuring from inside the code:

import torch, time

def report(tag: str):
    a = torch.cuda.memory_allocated() / 1024**3      # what tensors actually use
    r = torch.cuda.memory_reserved() / 1024**3       # what PyTorch reserved from the card
    p = torch.cuda.max_memory_allocated() / 1024**3
    print(f"{tag:24s} allocated={a:6.2f} GB reserved={r:6.2f} GB peak={p:6.2f} GB")
    torch.cuda.reset_peak_memory_stats()

report("before load")

from transformers import AutoModelForCausalLM, AutoTokenizer

mid = "Qwen/Qwen3-8B"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(
    mid, torch_dtype=torch.bfloat16, device_map="cuda",
    use_safetensors=True,          # insist on the safe format
)
report("after weights load")

ids = tok("Explain how the KV cache works", return_tensors="pt").to("cuda")
t0 = time.perf_counter()
out = model.generate(**ids, max_new_tokens=256, do_sample=False)
dt = time.perf_counter() - t0
report("after generate")

n = out.shape[-1] - ids["input_ids"].shape[-1]
print(f"generated {n} tokens in {dt:.2f}s = {n/dt:.1f} tok/s")
print(torch.cuda.memory_summary(abbreviated=True))

5.3 The numbers that matter, and what they mean

Number	Where from	How to read it
VRAM used	nvidia-smi	Above 95% you risk OOM as context grows. Reduce context or quantize the KV cache
GPU utilization	nvidia-smi	High is not necessarily good. During decode this reads high even while the card waits on memory
TTFT	Measure it yourself	Time to first token, reflecting prompt-reading speed. This is the number users feel most
tok/s (decode)	Measure it yourself	Typing speed of the answer, bounded mainly by bandwidth
Total throughput	Runtime metrics	Everyone’s tok/s combined — the number that matters when serving many
KV cache usage	vLLM /metrics	Regularly above 90% means rejections or long queues ahead. Cut context or add cards
Power draw	nvidia-smi	Well below TDP under full load means the bottleneck is memory, not arithmetic
Throttle reasons	nvidia-smi	SwThermalSlowdown means it is too hot — fix cooling, not software

5.4 Benchmarking without fooling yourself

Mistakes that produce fictional numbers:

• Timing the first run, which includes model loading and kernel compilation
• Measuring once and concluding
• Using a 10-token prompt and calling it fast, when real prompts are far longer
• Measuring one user and multiplying by the number of users
• Not separating TTFT from decode, when the two bottleneck in different places

The right way:

1. Discard three warm-up runs before measuring
2. Use prompts as long as the real workload — say 1,000–4,000 tokens
3. Measure at least ten runs and report median and p95, not the mean
4. Sweep concurrency at 1, 2, 4, 8, 16 and find where latency turns upward
5. Record peak VRAM across the whole test, not the value at the start
6. Repeat after 20 minutes of heat; numbers usually fall from their cold-start figures

A script that separates TTFT from decode and sweeps concurrency:

import asyncio, statistics, time, httpx

BASE = "http://localhost:8000/v1/chat/completions"
PROMPT = "Summarise the key points of the following as a list:\n" + ("sample content " * 400)

async def one(client) -> tuple[float, float, int]:
    t0, first, n = time.perf_counter(), None, 0
    async with client.stream("POST", BASE, json={
        "model": "Qwen/Qwen3-8B",
        "messages": [{"role": "user", "content": PROMPT}],
        "max_tokens": 256, "temperature": 0, "stream": True,
    }, timeout=180) as r:
        async for line in r.aiter_lines():
            if line.startswith("data: ") and "[DONE]" not in line:
                if first is None:
                    first = time.perf_counter() - t0       # TTFT
                n += 1
    total = time.perf_counter() - t0
    decode = n / (total - first) if first and total > first else 0
    return first, decode, n

async def sweep():
    async with httpx.AsyncClient() as c:
        await asyncio.gather(*[one(c) for _ in range(3)])   # warm-up, discarded
        for users in (1, 2, 4, 8, 16):
            res = await asyncio.gather(*[one(c) for _ in range(users)])
            ttft = [r[0] for r in res]
            decode = [r[1] for r in res]
            print(f"users={users:2d} "
                  f"TTFT med={statistics.median(ttft):.2f}s "
                  f"p95={sorted(ttft)[int(len(ttft)*0.95)-1]:.2f}s "
                  f"decode med={statistics.median(decode):.1f} tok/s "
                  f"total={sum(decode):.0f} tok/s")

asyncio.run(sweep())

What the results are telling you

As concurrency rises, if total throughput climbs while per-user tok/s falls, that is healthy — the system is batching well. If total throughput stops climbing, you have hit the card’s ceiling, and every additional user only makes everyone slower. That point is your machine’s real capacity, and it is the number to plan against — not the one on the brochure.

What this chapter settles

VRAM is not just the model. The KV cache scales with context length times concurrent users, and that is where systems break on launch day. Benchmark with realistic prompt lengths, separate TTFT from decode, report median and p95, and sweep concurrency until you find the ceiling.

The next chapter is about choosing a model for the job — starting from your constraints rather than from a leaderboard.