VerChol -- Grammar-First Tokenization for Agglutinative Languages

Tokenization is the foundational step in all large language model (LLM) pipelines, yet the dominant approach Byte Pair Encoding (BPE) and its variants is inherently script agnostic and optimized for English like morphology. For agglutinative languages a typological class encompassing the Dravidian family (Tamil, Kannada, Telugu, Malayalam), Turkic languages (Turkish, Azerbaijani, Uzbek), Uralic languages (Finnish, Hungarian, Estonian), Korean, Japanese, Swahili, Basque, and others, a single word may encode root, tense, aspect, person, number, gender agreement, case, and postpositions into one orthographic unit. Statistical tokenizers fragment these words into byte pair chunks that sever morpheme boundaries and inflate token counts.

💡 Research Summary

The paper “VerChol – Grammar‑First Tokenization for Agglutinative Languages” addresses a fundamental inefficiency in modern large‑language‑model pipelines: the reliance on statistical sub‑word tokenizers such as Byte‑Pair Encoding (BPE). While BPE works well for English‑like morphologies, it ignores morpheme boundaries and therefore inflates token counts for agglutinative languages, where a single orthographic word may encode a root, tense, aspect, person, number, gender, case, and postpositions. The authors propose VerChol, a language‑parametric tokenizer that follows a four‑tier pipeline built on linguistic principles rather than corpus statistics.

The pipeline consists of:

1. Tier 0 – Whole‑word vocabulary lookup – a pre‑constructed list of valid inflected forms (derived from a root dictionary, suffix catalog, and script‑specific syllable inventory). If a word is found, it is emitted as a single token.
2. Tier 1 – Rule‑based morphological decomposition – for out‑of‑vocabulary words, a deterministic morphological analyzer splits the surface string into contiguous root‑plus‑suffix spans, guaranteeing 100 % round‑trip fidelity.
3. Tier 2 – Syllable segmentation – when morphological analysis fails, language‑specific phonotactic rules break the word into valid CV/CVC (or language‑specific) syllables.
4. Tier 3 – Character fallback – the last resort emits individual characters.

Crucially, the tier logic is language‑agnostic; only four language‑specific modules (root dictionary, suffix catalog, phonology/sandhi rules, and script table) need to be swapped to support a new language.

The authors evaluate VerChol on Tamil, a representative agglutinative language, using the full Tamil Wikipedia corpus (774 MB, 30.5 M word tokens, 1.85 M unique types). They construct a 32 991‑token vocabulary (VerChol‑32K) without any training compute—everything is derived from linguistic resources. On the set of 483 313 unique words that appear at least three times, VerChol‑32K achieves a fertility of 1.86 tokens per word, compared to 2.85 for a SentencePiece BPE model trained on the same data (16 K vocab) and 3.52 for a production Indic‑optimized BPE (68 K vocab). This translates to 35 % fewer tokens than standard BPE and 47 % fewer than the Indic‑optimized baseline, despite using roughly half the vocabulary size. A smaller 16 K VerChol model (VerChol‑16K) yields a similar fertility (1.89), demonstrating that the morphological engine—not vocabulary size—is the primary source of compression.

Tier distribution analysis shows that 35.5 % of words are resolved at Tier 0, 55.5 % at Tier 1, and only 9 % require syllable or character fallback. Thus, over 90 % of tokens are linguistically meaningful morphemes or syllables, and the system generalizes to unseen word forms because any known root combined with any known suffix is decomposed correctly, regardless of corpus frequency.

The paper also outlines an adaptation framework for Turkish, Finnish, Korean, and Swahili. For Turkish, the main challenge is vowel harmony; the solution is to enumerate all allomorphic suffix variants in the suffix catalog. Finnish requires handling 15 grammatical cases and consonant gradation, which are encoded as phonological rules in Tier 1. Korean uses Jamo‑level syllable blocks. The authors claim that, given existing linguistic resources (root lists, suffix tables), a functional VerChol module for each language can be built within a week of engineering effort.

Limitations are acknowledged: rule‑based systems need manual updates for neologisms and foreign loanwords, and the current evaluation focuses solely on token‑level efficiency (fertility) without measuring downstream LLM performance (training speed, perplexity, downstream task accuracy). Future work is proposed to automate rule extraction, enable dynamic vocabulary expansion, and benchmark the impact of VerChol tokenization on actual language‑model training and inference.

In summary, VerChol demonstrates that a grammar‑first, morphology‑aware tokenizer can dramatically reduce token counts for agglutinative languages, offering a language‑agnostic architecture that only requires swapping a small set of linguistic modules. This work provides a compelling alternative to purely statistical sub‑word tokenizers and opens a path toward more efficient multilingual LLM pipelines for the many billion speakers of agglutinative languages.