Chapter 4 of 5 · Fine-Tuning Language Models — When to Do It, How, and How to Know It Worked

Evaluating a fine-tune: your own test set, memorising, and forgetting

Without an evaluation set you cannot tell whether fine-tuning worked. This chapter covers splitting training and test data, memorising instead of learning, forgetting existing abilities, the Thai benchmarks available, and Thai-specific traps such as answers that drift into English.

OpenAI puts it plainly: only invest in fine-tuning after setting up evals. Without numbers before and after, you cannot say whether training made things better, worse or no different.

4.1 Split training and test data from the start

OpenAI recommends splitting the dataset into training and test portions: the training set teaches, the test set measures, and the model must never see the test set during training.

Common mistakes:

Near-duplicates. If the same customer message, differing only in a closing word, sits in both sets, scores look better than they are.
The test set must look like real work, including hard cases and messages with typos or mixed languages, not just the tidy examples.
Measure the base model first, on the same test set with your best prompt, so you have a number to compare against.

4.2 Memorising instead of learning (overfitting)

OpenAI describes overfitting as a model that improves early on but then memorises the dataset instead of learning generalisable knowledge. Signs to watch:

scores on the training set keep rising while scores on the test set stall or fall;
answers become repetitive — OpenAI advises that if the model becomes less diverse than expected, decrease the number of epochs by 1 or 2.

4.3 Forgetting what it knew (catastrophic forgetting)

Training a new task can make a model worse at old ones. Luo et al. found that this forgetting is generally observed in models from 1B to 7B parameters, and within that range it gets more severe as models get larger.

The study comparing LoRA with full fine-tuning (chapter 3) found LoRA better preserves abilities outside the target task. A real example from ThaiLLM-8B’s model card, continued-trained from Qwen3-8B-Base on Thai data: several Thai scores rose, but maths in the M6 exam fell from 0.4090 to 0.2727, while English MMLU dropped only 0.0126. Forgetting can hit one area and stay invisible in an average.

The way to check is two test sets: one for the task you taught, another for general abilities you still need, such as answering general Thai questions correctly or following instructions.

4.4 Thai benchmarks available

Benchmark	What it measures	Size
ThaiExam	Thai multiple-choice exams: ONET, IC, TGAT, TPAT-1, A-Level	590 questions
MT-Bench Thai	Multi-turn conversation in 9 categories, including Thai cultural knowledge	91 questions
IFEval-TH	Instruction following, translated with GPT-4 and corrected by people	215 items
OpenThaiEval	Thai exams, natural language inference and regional knowledge	Not stated on the pages we checked; see the Typhoon-S paper

These measure general ability, not your task. Use them to check for forgetting as in 4.3, but your main test set must come from your own real work.

The QLoRA paper also warns that current chatbot benchmarks are not trustworthy for accurately measuring chatbot performance, so treat chatbot benchmark scores as supporting evidence, not the verdict.

4.5 Thai-specific traps

Drifting into English. The Typhoon 2 paper found that English-first or English–Chinese models have a high tendency to switch language mid-answer when told to reply in Thai. Your test set should check that answers stay in Thai.
Counting words needs a segmenter. Thai has no spaces between words, so word-based metrics such as ROUGE, or IFEval instructions like “answer in at most N words”, need a Thai word segmenter such as PyThaiNLP, or the numbers will be wrong.
Translated benchmarks read like translations. SEA-LION’s instruction-following and conversation benchmarks were localised by native speakers to avoid translationese and lost cultural context, and some English instructions, such as changing capitalisation, mean nothing in Thai script.

4.6 A hypothetical example: a great score that fails in use

The shop from earlier chapters trains its sorting model and gets a very high score on its test set. In real use, it misfiles messages often.

Looking back, the test set had been sampled from the same period as the training set, so the same customers writing several times in similar words appeared in both. The model had just memorised them. The fix is to split the test set by time or by customer, so its messages are ones the model has truly never seen.

4.7 After this chapter

Separate the test set from the training set, remove near-duplicates, and if the same people appear many times, split by person or by period.
Measure the base model with your best prompt first, and record the score as your baseline.
Add a second test set for general abilities that must not be lost.
Check that answers stay in Thai, and use a Thai word segmenter for every word-counting metric.

The last chapter covers putting the model into real use.