Chapter 5 of 5 · Fine-Tuning Language Models — When to Do It, How, and How to Know It Worked

Deploying a fine-tuned model: merge the adapter, convert to GGUF, run with Ollama or vLLM

From adapter to a model you can run: merge the adapter into the base model, convert to GGUF and shrink it with llama.cpp, import it into Ollama — which stopped accepting separate LoRA adapters in September 2026 — or serve the adapter directly with vLLM, plus the licence conditions when you distribute.

Training leaves you with a small adapter file that must be used with its base model. This chapter covers the two common routes: merge into one model and run it locally with Ollama, or serve the adapter directly with vLLM.

5.1 Merge the adapter into the base model

PEFT’s merge_and_unload() merges the adapter’s weights into the base model. You get an ordinary model that no longer depends on the adapter and has no adapter latency.

If you trained with QLoRA, do not merge onto the 4-bit model you trained with. The merged model would still be bitsandbytes 4-bit, which llama.cpp’s converter in the next step does not support. Reload the base model in 16 bits, then load the adapter and merge:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("<base-model>", dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, "./my-adapter")
model = model.merge_and_unload()
model.save_pretrained("./my-merged-model")
AutoTokenizer.from_pretrained("./my-adapter").save_pretrained("./my-merged-model")

The last line saves the tokenizer in the same folder, because the converter reads it from the model’s folder. Use the tokenizer you trained with (if it was not saved with the adapter, use the base model’s). PEFT’s docs also warn that merge_and_unload() is not an in-place operation: it returns a new model, so take the returned value as above.

5.2 Convert to GGUF and shrink it

GGUF is llama.cpp’s file format, used by Ollama and several other local runners. You need to download and build llama.cpp and install its Python requirements first. Then there are two steps:

Convert the Hugging Face model to GGUF with llama.cpp’s convert_hf_to_gguf.py, choosing a data type with --outtype, such as bf16.
Shrink it (quantize) with llama-quantize, for example to Q4_K_M.

python convert_hf_to_gguf.py ./my-merged-model --outfile my-model-bf16.gguf --outtype bf16
./build/bin/llama-quantize my-model-bf16.gguf my-model-Q4_K_M.gguf Q4_K_M

A size example from llama.cpp’s docs: Llama-3.1-8B takes 14.96 GiB in F16 and 4.58 GiB in Q4_K_M. The docs also warn that re-quantizing an already quantized file can severely reduce quality; always quantize from a 16- or 32-bit file.

Evaluate again after quantizing, with the test set from chapter 4, because quantization can lower quality, especially for tasks that need precision.

5.3 Running with Ollama: merge first

What changed in September 2026: Ollama no longer supports LoRA adapters. The ADAPTER instruction is gone from the Modelfile docs, and from Ollama v0.34.1 ollama create rejects a file containing ADAPTER. Older articles that put an adapter in the Modelfile do not work with current Ollama.

For GGUF, merge first (5.1), convert and quantize to GGUF (5.2), then import (Ollama’s docs also describe importing a safetensors folder directly; check the docs for your version):

FROM ./my-model-Q4_K_M.gguf

Save that as a file named Modelfile and run ollama create my-model in the same folder. Ollama’s docs stress that Ollama does not quantize GGUF models during import, so quantize first.

After importing, check the chat template with ollama show --modelfile my-model against the one you trained with (chapter 2); if it differs, set TEMPLATE in the Modelfile.

5.4 Serving the adapter with vLLM

If you run a server with vLLM, you need not merge: vLLM serves adapters alongside the base model directly, and several adapters on one base model.

vllm serve <base-model> --enable-lora --lora-modules my-task=./my-adapter

vLLM’s maximum adapter rank defaults to 16; if your adapter was trained with a higher rank, such as 32 or 64, add --max-lora-rank.

vLLM can also load adapters while the server runs, but its docs warn this should not be used in production unless it is an isolated, fully trusted environment.

5.5 Distributing a model: read the licence again

If you share or sell a fine-tuned model, the base model’s terms come with it:

Base model	What you must do when distributing (per the licences checked)
Llama 4	A fine-tuned Llama, or a model trained on Llama outputs, that you distribute must have a name beginning with “Llama”; include a copy of the licence and display “Built with Llama”
Gemma models before Gemma 4	Pass Gemma’s use restrictions on in your agreement with recipients, include a copy of the terms, mark modified files, and add a “Notice” file
Gemma 4, Qwen3.8-27B and other Apache 2.0 models	Follow Apache 2.0, such as including the licence and stating your changes
A model itself built on another model	Read the licences of both the model you use and the one underneath (chapter 3)

This table is not exhaustive and is not legal advice; read the full licence before distributing.

5.6 A hypothetical example: faster, but wronger

The shop from earlier chapters quantized its sorting model to 4 bits so it would run fast on the shop’s machine, and put it into use without re-testing. Later it found the “complaint” category, which had few examples, was being misfiled more often.

The lesson: every time you change the model’s form — merging, converting or quantizing — rerun the same test set and look at the score per category, not just the overall score.

5.7 The whole series as a checklist

Chapter	Check
1	You tried prompting seriously; the problem is format or behaviour, not frequently changing data
2	Data is in the same messages format used in production, with the base model’s chat template, written labelling definitions, and licences and provider terms that allow it
3	The base model’s Thai is good enough, both licence layers fit your use, and you follow the docs for the tool versions installed
4	A truly separate test set, a baseline from the base model, a forgetting check, and a check that answers stay in Thai
5	Merge before importing into Ollama, re-evaluate after quantizing, and meet the licence conditions when distributing