What Kinds of Runtime Are There?

4.1 The layers of the stack

[6] What the user sees   Open WebUI, LM Studio, ComfyUI, your own app
        ↑ speaks HTTP, usually an OpenAI-compatible API
[5] Server               Ollama serve, vLLM serve, llama-server, Triton
        ↑
[4] Engine               llama.cpp, vLLM, SGLang, TensorRT-LLM, ExLlamaV3, MLX
        ↑
[3] Framework            PyTorch, JAX, GGML
        ↑
[2] Compute libraries    CUDA / cuDNN / cuBLAS, ROCm, Metal, Vulkan
        ↑
[1] Driver and hardware  NVIDIA driver, GPU

The problems beginners actually hit live at layers 1–2, not 4–5: a CUDA version that does not match PyTorch, or a driver too old for a new card. When something breaks, work upward from the bottom.

4.2 The main runtimes, and when to use each

Runtime	File format	Strength	Use it when
Ollama	GGUF	One-command install, manages VRAM itself, OpenAI-style API	Getting started, single-user on your own machine, prototyping
LM Studio	GGUF, MLX	A window to click, no terminal needed	People not at home on the command line, trying many models
llama.cpp	GGUF	The most flexible; runs on almost anything, splits load across GPU/CPU	Constrained hardware, mixed card generations, fine control
vLLM	safetensors, AWQ, FP8	High throughput, continuous batching, prefix caching	Serving many users, real production workloads
SGLang	safetensors	Low latency, structured output, strong on repeated prompts	Agent work, forcing JSON output, prompts with large shared prefixes
TensorRT-LLM	Compiled engine	Fastest on NVIDIA	Very high volume, when the compilation overhead is acceptable
ExLlamaV3	EXL3	Very fast on a single consumer card	Squeezing maximum speed out of the card you own
MLX	MLX	Fastest on Apple Silicon	A Mac is your main machine
ONNX Runtime	ONNX	Cross-platform, good on CPU and embedded devices	Small models on customer machines, mobile, edge

What changed in 2026

Hugging Face TGI, once a popular production choice, moved to maintenance status in March 2026 with no new feature work. Existing deployments still run, but it is no longer the recommended starting point for new projects. The general lesson is worth keeping: this ecosystem moves fast, so design against a standard API and you can change runtimes without touching the application.

4.3 Running them for real

Ollama — the easiest place to start

# Install (Linux/macOS)
curl -fsSL https://ollama.com/install.sh | sh

# Pull and chat immediately
ollama run qwen3:8b

# See which models exist, and what each uses in VRAM while running
ollama list
ollama ps

Calling it through the OpenAI-style API on port 11434:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3:8b",
    "messages": [{"role":"user","content":"Summarise this paragraph as briefly as possible: ..."}],
    "temperature": 0.3
  }'

Changing the defaults with a Modelfile:

cat > Modelfile <<'EOF'
FROM qwen3:8b
PARAMETER num_ctx 8192          # context length
PARAMETER temperature 0.3
PARAMETER num_gpu 999           # layers to keep on the GPU (999 = all)
SYSTEM "Answer concisely and stay on the point."
EOF
ollama create my-assistant -f Modelfile

llama.cpp — the finest control

# Build from source with CUDA support
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j --config Release

# Download a model from Hugging Face
huggingface-cli download Qwen/Qwen3-8B-GGUF Qwen3-8B-Q4_K_M.gguf \
  --local-dir ./models

Serving it, with an OpenAI-style API on port 8080:

./build/bin/llama-server \
  -m ./models/Qwen3-8B-Q4_K_M.gguf \
  -ngl 99 \                      # layers pushed to the GPU; lower it if VRAM is short
  -c 8192 \                      # context length
  -np 4 \                        # concurrent slots
  --host 0.0.0.0 --port 8080 \
  --flash-attn \                 # saves attention VRAM
  -ctk q8_0 -ctv q8_0            # quantize the KV cache; a large saving at long context

# Measure your own machine
./build/bin/llama-bench -m ./models/Qwen3-8B-Q4_K_M.gguf -p 512 -n 128

vLLM — for serving many users at once

uv pip install vllm

vllm serve Qwen/Qwen3-8B \
  --port 8000 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \   # use up to 90% of VRAM, hold back the rest
  --kv-cache-dtype fp8 \            # roughly halves KV cache VRAM
  --enable-prefix-caching \         # reuse identical prompt prefixes
  --tensor-parallel-size 1          # how many cards share the computation

# Prometheus metrics
curl http://localhost:8000/metrics | grep -E "num_requests|gpu_cache_usage"

SGLang — agent work and JSON

uv pip install "sglang[all]"

python -m sglang.launch_server \
  --model-path Qwen/Qwen3-8B \
  --port 30000 \
  --context-length 16384 \
  --enable-torch-compile

Forcing an answer to match a schema:

curl localhost:30000/v1/chat/completions \
  -d '{
    "model":"default",
    "messages":[{"role":"user","content":"Extract the vendor and total from this receipt"}],
    "response_format":{
      "type":"json_schema",
      "json_schema":{"schema":{
        "type":"object",
        "properties":{
          "vendor":{"type":"string"},
          "total":{"type":"number"}},
        "required":["vendor","total"]}}}
  }'

Images and audio

Image generation with diffusers:

import torch
from diffusers import AutoPipelineForText2Image

pipe = AutoPipelineForText2Image.from_pretrained(
    "stabilityai/sdxl-turbo",
    torch_dtype=torch.float16,
    use_safetensors=True)        # insist on the safe format
pipe.to("cuda")
pipe.enable_model_cpu_offload()  # saves VRAM

img = pipe("a cat in a library", num_inference_steps=4).images[0]
img.save("out.png")

Transcription with faster-whisper:

from faster_whisper import WhisperModel

m = WhisperModel("large-v3", device="cuda", compute_type="float16")
segs, info = m.transcribe("meeting.mp3", language="th")
for s in segs:
    print(f"[{s.start:.1f}s] {s.text}")

4.4 The common standard that makes runtimes swappable

Nearly every runtime exposes an OpenAI-shaped API at /v1/chat/completions. If your application calls through that standard, moving from Ollama during prototyping to vLLM in production is a change of base URL, not a rewrite. Design it that way on day one.

from openai import OpenAI

# Only these two lines change when you switch runtime
client = OpenAI(
    base_url="http://localhost:11434/v1",   # Ollama
    # base_url="http://localhost:8000/v1",  # vLLM
    # base_url="http://localhost:8080/v1",  # llama.cpp
    # base_url="http://localhost:30000/v1", # SGLang
    api_key="not-needed-for-local",
)

resp = client.chat.completions.create(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Explain RAG in three lines"}],
    temperature=0.3,
    max_tokens=512,
)
print(resp.choices[0].message.content)

What this chapter settles

Choose a runtime by the job, not by popularity: Ollama to start and for single-user work, llama.cpp when hardware is tight, vLLM when several people are served at once, SGLang when the answer shape must be enforced. And whichever you pick, have the application speak the standard API so the runtime can be replaced without touching your code.

The next chapter is about measuring what a model actually consumes — the place where real systems most often break.