Fine-tuning Thai OCR for production: seven full fine-tunes that barely helped, a LoRA that works in practice, the guards built with it, and a leaky test set

This series is about AI research we actually did, not hypothetical examples. We do not name the project or the client, but every number comes from our own files, in three kinds:

• Recomputed from the raw result files while writing this article (September 2026).
• Read directly from a configuration or results file, such as dataset sizes and training settings.
• “According to our notes at the time”: not yet recomputed. We mark every one.

This chapter covers fine-tuning an open 2B Thai OCR model to read Thai business documents such as bank passbooks, passports, financial statements and shareholder-list forms, from the first evaluated round on 7 July 2026 to production on 15 July 2026. It also covers a mistake we found while preparing this article.

If fine-tuning, LoRA or test sets are new to you, start with our Fine-Tuning Language Models series.

1.1 The task and how we measured it

Data: 12,218 usable pages from 4,492 submitted jobs, split into 11,205 training pages, 305 validation pages, and several test sets. The split was by job: all pages from one job land in the same set. The set used in every full fine-tuning round was 98 “hard” pages.

Three metrics:

• CER, the share of characters read wrongly against the answer key; 0 means every character matches. We report the median over pages.
• Hard-glyph error rate, errors only on characters that are often misread: rare consonants, Thai digits, and the vowel and tone marks ำ ุ ู ่ ้ ๊ ๋.
• Badly broken pages, pages with CER of 0.5 or more.

First lesson: the answer key is noisy too.

• The answer key for the 98 pages was corrected by an AI model and spot-checked by a person, not typed from scratch by people.
• We had an AI model transcribe 12 pages twice, independently, without either pass seeing the other. The two versions differed by a median CER of 0.115. Both passes were the same model, so the real noise is probably higher.
• Our pass mark was a median CER below 0.10, which is stricter than two answer keys agree with each other. Numbers around 0.1 therefore say little about which model reads better.

1.2 Seven rounds of full fine-tuning

Rounds R1 to R8 (no R7 was actually trained) trained the whole language part of the model and left the vision part untouched, with a learning rate of 1e-5 for 2 to 3 epochs.

Round	What changed	Result
R1	34% synthetic pages and 61% the original model’s own readings	According to our notes at the time, many pages were cut off early; 21 of 98 pages had CER above 0.6
R2	Better synthetic data, such as HTML tables and empty table cells	According to our notes at the time, better than R1
R2 + repetition_penalty	A repetition penalty of 1.1 on every page	Worse on every metric
R3	1,500 more invented copies of forms	Worse; our notes at the time diagnosed that the model learned the invented templates
R4	48 real pages whose answers an AI model corrected and a person spot-checked, each included 4 times (about 4% of the training set)	Clearly better than R3
R5–R8	More real pages, random image damage (augmentation), near-duplicates removed	Up and down; R8 best of the group
Soup	The weights of R4 and R5 averaged	Failed: worse than both R4 and R5 on the 76 clean pages (see 1.4)

A mid-training checkpoint often beat the final model. In R5 and R6, the checkpoint saved partway through had a better median than the final model. In R8 the checkpoint was better on hard glyphs, but on the 98 pages the final model had a slightly better median (0.120 against 0.124). All these margins are small enough to be chance. The lesson: training longer did not mean training better.

1.3 The leaky test set: what we just found

While preparing this article, we compared a fingerprint (md5) of every test page image with every image in every training set. At least 22 of the 98 test pages are byte-identical to an image in at least one training set.

• Why: we split by job, but the same file had been submitted in several jobs, so it ended up on both sides. The 98 test entries themselves contain only 93 distinct images.
• The clearest effect: we once concluded that R8 had solved a watermark problem, based on a passbook cover with a watermark tiled across the page. That cover appears in the test set as two entries that are the same image, and the image was in R8’s training data 4 times. The model had seen it before.
• A leak does not always help: two forms with rows of empty checkboxes were in the training data of R4 to R8, yet the R8 mid-training checkpoint still looped until it broke on both (CER 0.97), while the original model did much better (CER 0.50 and 0.46).
• 22 is a minimum: we only caught byte-identical files. The same picture saved again as a different file would not be caught.
• The public benchmark did not leak into our training data: none of the 140 images in the public benchmark used in 1.5 appear in our training sets. We cannot say whether the base model’s makers trained on it.

The project’s own notes say never to train on the test set. The rule was right; splitting by job did not actually enforce it.

1.4 Recomputed on the 76 clean pages

The numbers in brackets are the original values on all 98 pages. Every row used the same loop-stop and retry guards at reading time, except the R2 + repetition_penalty row, which ran without guards.

Model	Median CER	Hard-glyph error rate	Badly broken pages
Original model + guards	0.092 (0.124)	0.0365 (0.0648)	2 (10)
R2 + repetition_penalty, no guards	0.116 (0.169)	0.0906 (0.1617)	5 (18)
R3	0.118 (0.152)	0.0524 (0.1060)	2 (14)
R4	0.098 (0.132)	0.0645 (0.0792)	2 (5)
R6 mid-training checkpoint	0.091 (0.132)	0.0363 (0.0543)	2 (4)
R8 final	0.089 (0.120)	0.0355 (0.0745)	0 (4)
R8 mid-training checkpoint	0.087 (0.124)	0.0263 (0.0580)	0 (3)
Soup R4+R5	0.127 (0.135)	0.0997 (0.0977)	6 (6)

How to read it:

• Page by page against the original model, no round wins beyond chance. The closest, the R6 checkpoint, was better on 43 pages and worse on 32. The R8 checkpoint was better on 36 and worse on 37; R4 better on 30 and worse on 40.
• Two things look better: badly broken pages fell from 2 to 0, and the hard-glyph error rate fell from 0.0365 to 0.0263. But 2 against 0 is too few to conclude anything, and the hard-glyph rate is a single pooled number we have not tested for chance.
• Removing leaked pages also hid a weakness. All 3 badly broken pages of the R8 checkpoint on the 98 pages are among the leaked ones, including the two checkbox forms from 1.3. So the 76-page table cannot show that fine-tuning created a new way to fail. Our notes at the time put it as fine-tuning “trades failure modes — not strictly better”.
• What you compare against matters. At the time we concluded that R4 was 5 times better than the original model, but that compared it with the original model without guards. Given the same guards, the original model had the better median (0.124 against 0.132 on 98 pages).
• More synthetic data did not help. Adding 1,500 synthetic forms in R3 made things worse (0.118); 48 real pages in R4 did not (0.098). We have not tested whether that difference is beyond chance.

1.5 Full fine-tuning made the model forget

We measured general ability with a public Thai OCR benchmark of 7 tasks with 20 images each, comparing the original model with the R8 mid-training checkpoint, using readings produced with the same settings and scored with the same metric.

• Full-page OCR: mean CER went from 0.308 to 0.597, but R8 was better on 8 images and worse on 12. A few badly broken images drove the mean up.
• Handwriting: recall fell from 0.885 to 0.707, worse on 14 images and better on 2. This is the clearest result.
• Document parsing and table parsing: both worse.
• Key information extraction was unchanged; two other tasks improved slightly, within what chance could explain.

Our notes at the time said “worse on 5 of 7 tasks, better on none”. Recomputed with this metric it is worse on 4, unchanged on 1 and slightly better on 2. The main conclusion stands: our task barely improved, and general ability was lost. So we changed approach.

1.6 Switching to LoRA (R9)

R9 left the model’s own weights untouched and trained a LoRA with rank 32, alpha 64 and dropout 0.05 on all 7 kinds of linear layer in the language part, with the vision part still frozen, a learning rate of 1e-4, and 2 epochs (v1 trained for 1).

The R9 versions were not measured on the 98 pages; they were measured on 6 specialised test pages, passports, and the public benchmark. The results on the specialised pages and passports, and the truncated answers below, are according to our notes at the time. The example counts and shares, and which pages were in training, we re-checked against the actual training sets.

The share of specialised data. The training set mixed general pages with specialised ones, such as hard-to-read shareholder-list forms.

• v1: only 5.4% specialised (12 of 221 examples); none of the 6 specialised test pages improved, and some got worse.
• v2: raised to 26.8%, but mixed results, 2 pages better and 2 worse. The cause: the answers we had a model generate were capped at 1,536 tokens, so 67% of them were cut short.
• v3: the same share, with answers regenerated under a 4,096-token cap; 4 of 6 pages better and none worse. But we have now found that 2 of those 6 pages were in v3’s training data, and both are among the pages that improved. Of the 4 pages not in training, 2 improved, 1 stayed the same, and on 1 the original model already read well.

The number of distinct examples may matter more than repetition. Passports were the other specialised task.

• v5: 5 passports, each repeated 4 times (12.2% of the training set). The trained passports were read correctly 5 of 5 times, unseen ones 0 of 2, and the model invented an answer once. It memorised rather than learned.
• v7: 15 passports, each repeated twice (17.2% of the training set). Unseen passports improved on 2 of 4.

From v5 to v7 both the number of distinct passports and their share went up, so we cannot tell which one caused the change, and the test samples are tiny (2 and 4 passports). This is a direction, not proof.

v7 used 174 examples and trained for 22 steps in 17 minutes.

v7 on the public benchmark, against the original model with the same settings and metric:

• No task got worse beyond chance.
• Handwriting and key information extraction improved slightly.
• Full-page OCR mean CER went from 0.342 to 0.294, but v7 was better on only 6 images and worse on 9, so we cannot say it really improved.

Our notes said it “passed”. That is right in the sense of no regression, but it does not mean better at everything. Where v7 is clearly better is something this benchmark does not measure; see 1.7.

Before launch, v7 and the original model both identified the document type correctly for 34 of 37 documents, and, according to our notes at the time, got rule-based fields right on 31 of 51 against 30 of 51. The original model ran on without stopping on 4 documents (see 1.7).

Note: the public-benchmark readings in 1.5 and in this section came from separate runs with different length caps and loop guards, which is why the original model scores differently (0.308 and 0.342). Never compare numbers across the two.

1.7 In real work: v7 finishes, the original model often does not

The benchmarks in 1.4 to 1.6 measure how accurately characters are read. But the original model’s big problem in real work was not stopping. It kept repeating the same text until it produced tens of thousands of characters of garbage; the text of that page could not be trusted, and the whole job slowed down. Here v7 is clearly different from the original model.

• 19 passports: the original model finished reading on its own for 2; v7 for 17. According to our notes at the time, with the guards added all 19 finished. A caution: at least 14 of the 19 were in v7’s training data; on unseen passports, according to our notes at the time, 2 of 4 improved.
• 37 real documents before launch: both versions identified the document type correctly for 34. But the original model looped into 12,926 to 44,097 characters on 4 documents: 2 shareholder-list forms and 2 passports. v7 read all 4 at a normal length (854 to 3,800 characters). Note that in this comparison v7 had the guard and the original model did not, which according to our notes at the time was how the live system was set up. The guard stepped in on only 1 of the 4; on the other 3, v7 finished on its own.
• 25 randomly chosen real documents after launch: all 25 finished. Called directly with no guard, the original model ran past 20,000 characters on 1; v7 on none.
• The day the system switched back to the original model by accident: according to our notes at the time, in September a test wrote a value into the live system’s settings file, and the system read with the original model for 46 minutes. Only test jobs ran in that window, no client jobs. Normal pages took 120 to 150 seconds instead of the usual 20 to 30, and a test set of 13 files ended with 11 done and 2 failed. After switching back to v7, with the same code and the same files, all 13 finished and none timed out or were killed. But this was not a planned experiment: the GPU was 94% busy while the original model ran and idle during the v7 run, so it is supporting evidence, not proof.

In short: on the public benchmark v7 “did not regress”, but in real work v7 finishes where the original model often runs on, and that is what made the system usable. Character accuracy alone is not enough to decide which model is ready for production.

1.8 Merging and going to production

All of this section is according to our notes at the time.

• Merge: the LoRA was merged into the base model to make one model (the same step as chapter 5 of the Fine-Tuning series), then served with vLLM.
• A way back: the original model was kept running alongside, ready to switch back to at once.
• A trap: starting two vLLM containers at the same moment made both compute a negative cache size and abort. Start them one at a time.
• But “no loops” was wrong. In September v7 looped on one passbook with a watermark tiled across the page every time we tried (3 of 3, the same file submitted three times). That is the same passbook layout as the page we once called a solved watermark problem in 1.3. No problem in 25 documents does not mean no problem. Section 1.9 describes how we dealt with it.
• The other checks stay. The project’s own notes say v7 improved reading but did not make it possible to drop the validators or the detector for invented text.

1.9 Guards developed alongside v7

v7 loops far less than the original model, but it can still loop. The second half of the work was developing the guards further, built and measured on v7 itself.

Starting point (July): the first guards came from the full fine-tuning rounds (the same ones used in the 1.4 table): check whether the end of the output is looping, and if it loops or does not finish, retry with temperature 0.2 and then 0.4. According to our notes at the time, a light loop check with these retries let v7 finish all 19 passports.

September’s lesson: measure guards on the model you actually run. According to our notes at the time, a plain re-read often looped again; on one page two reads gave exactly 61,627 characters each, although the result can change when run in a different batch (below, 2 of 4 pages still looped after a re-read). The “re-read fixes 9 of 9” figure we had recorded was measured on a different reading model, not v7.

Escape routes measured on 4 pages where v7 looped (2 of them are the same document):

• Re-read with the same prompt: still looping on 2 of 4.
• Re-read at temperature 0.3: still looping on 2 of 4.
• Limit output to 1,536 tokens: no longer too long, but it only cut the loop off rather than fixing it; according to our notes at the time, one page was still 83% repeated text.
• Re-read with a prompt asking for structured output: normal text on all 4.

A phrase-loop detector. The old checks only caught a single repeated character or repeated lines, but v7 sometimes repeats a whole sentence within one line. We built a new detector and measured it on 9,032 pages from 5,254 jobs before switching it on. According to our notes at the time, it flagged 51 pages below the old length limit; we opened every one, and all were real loops except 2 borderline cases. Account-statement pages, which look repetitive but are real data, were never flagged (0 of 12).

Stop continuing when a loop is found. Previously, when output ran past one chunk, the system asked the model to keep going for several more chunks even while it looped, so a looping page took 700 to 920 seconds and was then killed. According to our notes at the time, after this fix and a follow-up one (phrase loops skip the temperature retries), a looping page took about 90 seconds plus 30 to 90 seconds to re-read, and the 13 files that used to loop (6 real documents, some submitted more than once) all finished.

repetition_penalty only on looping pages. In 1.2, a repetition penalty on every page made every metric worse. On v7 we tried the same setting only on pages the detector flagged as looping, measured on 12 looping pages and 20 normal ones:

• Pages still looping: 5 with no penalty, 2 at 1.1, 1 at 1.2.
• Median time on the looping pages: 54 seconds, down to 19 at 1.1.
• At 1.2, according to our notes at the time, shareholder-list tables lost cells, so we chose 1.1.

The live system is therefore a ladder: when a loop is detected, re-read with a penalty of 1.1; if it still loops, use the structured-output prompt; if that fails, have a fallback reader read it; and if everything loops, cut the looping part off. According to our notes at the time, replaying the same files that used to loop, the penalty fixed 6 of 9 pages, including the watermarked passbook cover from 1.8, the structured prompt fixed the other 3, all 13 files finished, and one job went from 182 to 209 seconds down to 105 to 108. Pages that do not loop still send exactly the same request, byte for byte, so they are not affected.

What is still not solved, according to our notes at the time:

• Before the penalty step, some passbook terms-and-conditions pages had no reader that gave a stable result. The penalty step fixed that page on one replay, but whether it stays fixed still needs watching.
• The fallback reader has invented a whole page of text.
• The answer is to flag such pages for a person to check, not to trust them automatically.
• We once found 39 jobs where some pages had been read by another reading model without anyone knowing. The system now records and flags every page that was read by anything other than v7.

1.10 A checklist from this work

What to do	Lesson from
Measure how noisy the answer key is before setting a pass mark, and say clearly who or what made it	1.1
Split test sets by file content (a hash), not by job or file name, and re-check before trusting a result	1.3
After removing leaked data, check what failures the removed pages were hiding	1.3, 1.4
Compare against the original model with the same reading-time helpers	1.4
Look at paired, page-by-page results, not only the median or mean	1.4, 1.5
Measure general ability every round to catch forgetting	1.5
Be wary of adding large amounts of synthetic data	1.2, 1.4
Check answers generated by a model for truncation or broken format	1.6
Look at the number of distinct examples, not just their share	1.6
Measure real work too, such as whether reading finishes and how long it takes, not only character accuracy	1.7
Keep watching real documents after launch, not just the first batch	1.8
Develop and measure guards on the model you actually run, and apply strong fixes only to pages that have the problem	1.9

Where the numbers come from

• All numbers come from the project’s experiment files and notes, which we do not publish because they contain data from client documents.
• On 25 September 2026 we recomputed from the raw files: the duplicate-file check, the table in 1.4, the page-by-page comparisons, the public-benchmark results in 1.5 and 1.6, and the example counts for each version.
• Numbers marked “according to our notes at the time” have not been recomputed: for some the raw outputs were not kept, and others we have not re-scored yet.