Measuring Thai Text Programmatically: Why .split() and crfcut Lie to You — and How the "Wak" Saves the Day

If you have ever written code that analyzes English text, you carry two habits with you: counting words by splitting on spaces, and counting sentences by looking for periods. The moment you meet Thai, both habits betray you — and the cruel part is that they betray you silently. Nothing crashes. The numbers look plausible. They are simply all wrong.

This article comes from a real scar on this very website: we built a system that measures how “natural” our AI-written Thai articles read compared with news written by human Thai journalists. Before we could measure anything at all, we had to answer the most basic question correctly — what is the unit of Thai text, exactly?

Trap 1 — .split() cannot count words, because Thai has no spaces between them

Thai is written as an unbroken run of characters. There are no spaces between words, so len(text.split()) — perfectly fine in English — counts an entire clause as a single “word.”

We hit this ourselves. The first time we compared article lengths across the two languages, the system reported that the Thai version was dramatically “shorter” than the English one, despite carrying the same content: the English side was counting words, the Thai side was counting clauses.

The correct tool is PyThaiNLP’s word tokenizer:

from pythainlp import word_tokenize

text = "ภาษาไทยเขียนติดกันทั้งประโยค"
print(len(text.split()))          # 1  <- wrong
print(len(word_tokenize(text)))   # 6  <- ["ภาษาไทย", "เขียน", "ติดกัน", ...]

The default engine, newmm, is dictionary-based and fast enough for production use.

Trap 2 — crfcut segments sentences fine, until English words show up

The next problem runs deeper: Thai has no sentence-ending punctuation. No period, nothing that marks where a sentence stops. PyThaiNLP offers a sentence segmenter called crfcut — a CRF model trained on how humans insert spaces. It sounds great, and it hides two traps stacked on top of each other.

First, a silently-failing dependency. crfcut requires the python-crfsuite package. If it is missing, PyThaiNLP does not raise an error — it quietly falls back to whitespace splitting. The result is “one sentence per word,” and every statistic downstream is garbage with nobody the wiser.

Second, real text always contains Latin tokens. Thai tech articles are full of English product names and terms. We measured crfcut’s behavior on real news paragraphs: on pure Thai text it found 3 of 3 sentence boundaries correctly, but the moment an English product name appeared in the paragraph, accuracy fell to 1 of 4. Masking the Latin tokens before segmenting didn’t save us either — it started inventing boundaries inside product names, splitting “Gemini 3.6” away from “Flash.”

The most expensive lesson: it was not just wrong, it was flatteringly wrong. The phantom segments crfcut invented once gave one of our articles a short-sentence ratio of 0.472 — the best score in the whole test set. Measured honestly, the real figure was 0.258. A pretty number from a broken tool is far more dangerous than an ugly number from a working one.

The way out — measure the unit Thai actually has: the wak

Step back and ask a different question: what boundary does a Thai writer actually mark in their text? There is exactly one answer — the wak (วรรค), the space deliberately typed to break the reading rhythm. Instead of forcing a “sentence” concept onto a language that never marks one, we measure in wak units. And it turns out no model is needed at all:

import re

# split only on spaces flanked by Thai characters on BOTH sides
THAI = "ก-๛"
CLAUSE_SPLIT = re.compile(rf"(?<=[{THAI}]) (?=[{THAI}])")

def thai_clauses(paragraph: str) -> list[str]:
    return [c.strip() for c in CLAUSE_SPLIT.split(paragraph) if c.strip()]

The lookbehind/lookahead pair is the clever bit: the spaces inside “Gemini 3.6 Flash” or “$0.30 per million tokens” are not split, because at least one side is not a Thai character. Product names and figures survive intact with zero exception lists.

What can you actually measure with wak units?

Once the unit is trustworthy, three simple metrics fall out:

1. Short-clause ratio — the share of clauses that are 10 words or fewer (counted with newmm)
2. Coefficient of variation (CV) of clause length — higher means more varied rhythm
3. Average words per clause

A number in isolation means nothing until real writing sets the baseline. So we built a reference corpus from 29 human-written Thai IT-news articles across 6 Thai outlets (keeping only aggregate statistics, never the original text). The result is the chart at the top of this article:

That first row is the sharpest gap we have ever measured: 4 of our 5 AI-written Thai articles scored below every single human article in the reference corpus. In plain language: AI writes Thai in long, evenly-paced clauses, while human journalists constantly alternate short and long. What we used to vaguely call “Thai rhythm” is now a reproducible number, not a feeling.

Bonus: Thai spacing rules compile down to regex too

Thai spacing follows established conventions from style guides and the Royal Society’s orthography rules — and most of them can be checked with plain regex:

RULES = [
    ("space required after mai yamok ๆ",  r"ๆ[^\s\)\]\.,:;]"),
    ("space required after a comma",      r",(?!\d{3}\b)\S"),   # 1,000 exempt
    ("no spaces inside parentheses",      r"\(\s|\s\)"),
    ("digits and % written together",     r"\d\s%"),
]

Four lines catch the most common Thai typing errors without a single LLM call.

Three lessons worth keeping

One — the tool’s unit is not the language’s unit. Do not force “sentences” onto a language that never marks them. Measure the wak the writer actually typed.

Two — tools that fail silently are more dangerous than tools that fail loudly. Both the crfcut dependency fallback and the phantom segments that improved a score prove the same point: a measurement system must be most suspicious of its prettiest numbers.

Three — thresholds must come from measurement, not intuition. Our first target bands were calibrated on just 5 samples and two of them pointed the wrong way; only measuring against 29 real articles revealed it. And read results humbly: real journalism itself fails some of these bands, so failing one means “worse than most real journalism on this axis,” not “unusable.”

None of this is theory — it runs every day inside the content engine of the website you are reading, as the ruler that decides which drafts pass and which go back for another round. If you are building your own Thai NLP system, we hope our scars save you from earning the same ones.

Sources

1. PyThaiNLP — the open-source Thai NLP library
2. PyThaiNLP on GitHub