AdaptBPE: From General Purpose to Specialized Tokenizers

Subword tokenization methods, such as Byte-Pair Encoding (BPE), significantly impact the performance and efficiency of large language models (LLMs). The standard approach involves training a general-purpose tokenizer that uniformly processes all textual data during both training and inference. However, the use of a generic set of tokens can incur inefficiencies when applying the model to specific domains or languages. To address this limitation, we propose a post-training adaptation strategy that selectively replaces low-utility tokens with more relevant ones based on their frequency in an adaptation corpus. Our algorithm identifies the token inventory that most effectively encodes the adaptation corpus for a given target vocabulary size. Extensive experiments on generation and classification tasks across multiple languages demonstrate that our adapted tokenizers compress test corpora more effectively than baselines using the same vocabulary size. This method serves as a lightweight adaptation mechanism, akin to a vocabulary fine-tuning process, enabling optimized tokenization for specific domains or tasks. Our code and data are available at https://github.com/vijini/Adapt-BPE.git.

💡 Research Summary

The paper tackles the inefficiency that arises when a large language model (LLM) continues to use a static, general‑purpose subword tokenizer—typically a Byte‑Pair Encoding (BPE) tokenizer—after pre‑training. Because the merge list of a BPE tokenizer is learned once on a massive pre‑training corpus and then frozen, the token set often contains many low‑frequency “junk” tokens that are rarely useful for a specific downstream domain or language pair. When the test distribution diverges from the pre‑training distribution, these tokens increase the length of tokenized sequences, waste model parameters, and raise inference latency.

To address this, the authors propose AdaptBPE, a lightweight post‑training adaptation algorithm that reshapes an existing BPE tokenizer without altering the underlying LLM weights. The method works under a fixed “merge budget” N (the desired vocabulary size). Starting from the first N merges of the original tokenizer, the algorithm iteratively replaces low‑utility merges with higher‑utility alternatives drawn from the remaining merges, guided solely by statistics computed on an adaptation corpus.

The procedure can be summarized as follows:

1. Initialize: Take the full ordered merge list µ =