BBPE16: UTF-16-based byte-level byte-pair encoding for improved multilingual speech recognition

Multilingual automatic speech recognition (ASR) requires tokenization that efficiently covers many writing systems. Byte-level BPE (BBPE) using UTF-8 is widely adopted for its language-agnostic design and full Unicode coverage, but its variable-length encoding inflates token sequences for non-Latin scripts, such as Chinese, Japanese, and Korean (CJK). Longer sequences increase computational load and memory use. We propose BBPE16, a UTF-16-based BBPE tokenizer that represents most modern scripts with a uniform 2-byte code unit. BBPE16 preserves BBPE’s language-agnostic properties while substantially improving cross-lingual token sharing. Across monolingual, bilingual, and trilingual ASR, and in a multilingual continual-learning setup, BBPE16 attains comparable or better accuracy; for Chinese, it reduces token counts by up to 10.4% and lowers decoding iterations by up to 10.3%. These reductions speed up fine-tuning and inference and decrease memory usage, making BBPE16 a practical tokenization choice for multilingual ASR.

💡 Research Summary

This paper introduces BBPE16, a novel tokenizer designed to address inefficiencies in multilingual automatic speech recognition (ASR). The work identifies a key bottleneck in widely-used Byte-level Byte-Pair Encoding (BBPE) based on UTF-8 encoding. While UTF-8 BBPE is language-agnostic and covers all Unicode characters, its variable-length encoding (1-4 bytes per character) unnecessarily inflates token sequences for non-Latin scripts like Chinese, Japanese, and Korean (CJK). Longer sequences increase computational load, memory usage, and hinder efficient cross-lingual token sharing.

The proposed solution, BBPE16, shifts the internal representation from UTF-8 to UTF-16. UTF-16 encodes the vast majority of modern characters in the Basic Multilingual Plane (including Latin, CJK, Arabic, etc.) using a uniform 2-byte code unit. BBPE16 retains the language-agnostic benefits of standard BBPE but operates on this UTF-16 byte sequence. The processing pipeline involves converting input UTF-8 text to UTF-16, extracting the raw bytes (discarding the Byte Order Mark), applying the standard BPE merge algorithm on these bytes, and then converting the tokenized result back to UTF-8 for compatibility with existing systems.

The authors conduct extensive experiments to validate BBPE16 across multiple scenarios: monolingual (English, Korean), bilingual (English-Korean), trilingual (English-Korean-Chinese), and a multilingual continual-learning setup. Models are trained using a consistent E-Branchformer encoder and Transformer decoder architecture within the ESPnet framework, with differences only in the tokenizer (comparing standard BPE, UTF-8 BBPE, and BBPE16).

The results demonstrate several key advantages of BBPE16:

1. Performance Parity: BBPE16 achieves word error rate (WER) and character error rate (CER) comparable to or slightly better than both BPE and BBPE across all tested languages and settings, confirming no loss in recognition accuracy.
2. Enhanced Cross-Lingual Token Sharing: In the trilingual tokenizer analysis, BBPE16 dramatically increases the number of tokens shared between languages. While standard BBPE produced zero shared tokens for English-Korean, Chinese-English, and the three-language combination, BBPE16 generated 42, 55, and 42 shared tokens respectively. It also increased Korean-Chinese shared tokens from 95 to 573. This efficient sharing suggests better utilization of the model’s embedding space.
3. Improved Token Efficiency: BBPE16 reduces the average number of tokens per utterance, especially for Chinese text. In the trilingual setting, it achieved a 4.6% reduction compared to BBPE. In the continual-learning scenario on the Common Voice Chinese dataset, the token count reduction reached 10.4%. This directly translates to lower computational cost.
4. Faster Decoding: The reduced token sequence length leads to up to a 10.3% decrease in decoding iterations, speeding up inference.
5. Higher Vocabulary Coverage: BBPE16 utilizes a higher percentage of the allocated token vocabulary across all languages in a multilingual setup compared to BBPE, indicating more efficient use of the model’s parameters.

In the continual-learning experiment, where a pre-trained trilingual model is fine-tuned on additional domain-specific datasets (WSJ for English, Zeroth for Korean, Common Voice for Chinese), BBPE16 maintains performance parity while preserving its token efficiency gains.

In conclusion, BBPE16 presents a simple yet effective modification to the popular BBPE scheme by leveraging UTF-16’s uniform 2-byte representation. It successfully mitigates the sequence inflation problem of UTF-8 BBPE for non-Latin scripts, promotes significantly better cross-lingual token sharing, and reduces computational and memory overhead during both training and inference. The work positions BBPE16 as a practical and efficient tokenization choice for scalable multilingual and continual-learning ASR systems.