Hardware — Choosing a GPU and a Machine

Most people pick a GPU by its TFLOPS figure, which is the wrong number for inference. What decides which models you can run is VRAM. What decides how fast they answer is memory bandwidth. Raw compute matters for training and for chewing through long prompts — and there it matters a great deal — but not for generating tokens.

3.1 The three numbers, in order of importance

Number	What it decides	Why
VRAM (GB)	Which model sizes you can run	If a model does not fit, it either will not run or will swap against system RAM, which is ten to a hundred times slower. This limit is absolute
Bandwidth (GB/s)	How fast it answers	Generating a token requires reading the model’s weights, every time. Speed is bounded by memory, not arithmetic
Compute (TFLOPS)	Training speed, prompt-processing speed	Matters for fine-tuning, for long prompts, and for serving several users at once

A rough speed estimate:

peak tok/s ≈ bandwidth ÷ model size

For an 8B model at Q4_K_M, roughly 5 GB:

Hardware	Working	Theoretical
A card with 900 GB/s	900 ÷ 5	≈ 180 tok/s
A card with 300 GB/s	300 ÷ 5	≈ 60 tok/s
CPU with DDR5 at ~80 GB/s	80 ÷ 5	≈ 16 tok/s

In practice you get about 60–70% of the theoretical figure. This is why the same model on two cards with identical VRAM but different bandwidth runs several times faster on one of them, and why CPU-only inference stays slow even on a powerful machine.

3.2 Hardware tiers by budget and job

Tier	VRAM	Runs comfortably	Suits
Laptop / iGPU	Shares system RAM	1–4B at Q4	Learning, experimenting, short summaries and translation
Entry card	8–12 GB	7–8B at Q4, embeddings, Whisper	Students, developers starting out, small RAG work
Mid card	16 GB	14B at Q4, 8B at Q8, image generation	Serious solo development
High-end card	24 GB	32B at Q4, 14B at Q8, LoRA on 8B	Work that needs real quality; light fine-tuning
Consumer flagship	32 GB	32B at Q5–Q6, 70B at Q3 (tight), video generation	The heaviest single job a consumer card can carry
Professional card	48–96 GB	70B at Q4–Q8, serious fine-tuning	Small teams, multi-user serving
Unified memory	64–128 GB	70B and up, or large MoE, at small batch	Very large models that do not need high throughput
Data centre	80–192 GB each	Everything, including real training	Organisations with real traffic

The buying rule for AI work

VRAM comes first, always. A card with more VRAM but less speed still works; a faster card without enough VRAM does not work at all. Given a choice between a new 16 GB card and a one-generation-older 24 GB card, for LLM work take the 24 GB in almost every case.

3.3 How unified memory differs from ordinary VRAM

Unified-memory systems — Apple Silicon, or NVIDIA’s GB10 — share one pool between CPU and GPU. That buys very large capacity far below the price of data-centre cards, but bandwidth is significantly lower than the GDDR or HBM on a discrete card.

Property	Discrete card (GDDR/HBM)	Unified memory
Capacity per unit of cost	Low — even 32 GB is expensive	High — 100 GB and beyond is reachable
Bandwidth	Very high	Several times lower
Good for	Fast answers, many concurrent users	Very large models, MoE, small batches, work that can wait
Bad for	Models larger than capacity	Services needing high throughput or very low latency

Why MoE and unified memory suit each other

Unified memory is constrained by bandwidth, not capacity — and MoE touches only a fraction of its parameters per token. A 100B model with 5B active reads less memory per token than a dense 30B model, even though it occupies far more capacity. Machines like this can therefore run models a consumer card cannot reach, at a genuinely usable speed.

3.4 The rest of the machine

Part	Rule of thumb	Why
RAM	At least equal to total VRAM; 1.5–2× is better	Model files are read through RAM on load; too little and loading is slow or fails
CPU	Need not be powerful for pure GPU work	Matters for data prep, tokenizing, and when offloading layers to CPU
SSD	NVMe, 2 TB and up	Models run 5–140 GB each and accumulate fast; read speed shows up as load time
PSU	1.5× total TDP	Modern cards have large transient spikes; an exactly-sized PSU will drop the machine mid-job
Cooling	More important than people expect	Inference is hours of sustained full load; too hot and the card throttles itself
PCIe	x8 is enough for single-card inference	Matters when loading models and when running tensor parallel across cards

3.5 Multiple cards — the common misunderstanding

Two cards is not two times the speed.

Tensor parallel splits each layer across cards so they compute simultaneously. It genuinely speeds things up, but the cards must talk at every layer, so it wants a fast interconnect. Over ordinary PCIe you may see 1.2–1.5×, not 2×. It also needs a card count that divides the attention heads evenly — usually 2, 4 or 8.

Pipeline parallel gives each card a different group of layers and passes work along like a conveyor. It makes large models fit, but does not make them faster for a single user. Use it when the goal is capacity, not speed.

Mixing card generations works in some runtimes — llama.cpp is the most tolerant — but speed is capped by the slowest card, and vLLM and SGLang generally want identical cards.

The practical conclusion: one card with enough VRAM beats two that have to be joined, in nearly every case, except when the model is large enough that there is no alternative.

3.6 Buy or rent

Situation	Recommendation
Still learning, unsure it will be used for real	Rent by the hour, or use an API. Do not buy a card yet
A few hours a day	Renting is almost certainly cheaper once power and depreciation are counted
All day, every day	Buying pays back within 6–12 months — but run the real break-even first
Data must not leave the organisation	Buy, or use a private cloud. This is not a price question
Frequent fine-tuning	Rent a large card per run rather than buying one that idles most of the time
Prototyping for a client	API first, always. Move to self-hosting once you know which model actually works

A simple break-even:

break-even (hours) = machine price ÷ (rental cost per hour − electricity per hour)

For example, a machine at 135,000 baht, an equivalent rental at 35 baht/hour, electricity at 5 baht/hour:

135,000 ÷ (35 − 5) = 4,500 hours
   ≈ 6 months at 24 hours a day
   ≈ 3 years  at 4 hours a day

Do not forget the invisible cost: the time spent maintaining the machine, fixing drivers and updating the system. For someone working alone, that is usually a larger cost than the electricity.

What this chapter settles

Get the order right: VRAM decides whether it runs, bandwidth decides how fast, and TFLOPS matters mainly for training and long prompts. One sufficient card beats two joined ones. And before buying, compute the break-even from actual usage rather than from intent.

The next chapter covers the runtimes: what kinds there are, how they differ, and the real commands for each.