Bolmo: Byteifying the Next Generation of Language Models

Recent advances in generative AI have been largely driven by large language models (LLMs), deep neural networks that operate over discrete units called tokens. To represent text, the vast majority of LLMs use words or word fragments as the tokens, known as subword tokenization. Subword tokenization obscures fine-grained information, which is problematic, especially for scientific data - such as computer code or biological sequences - where meaning depends on the individual characters. Models that instead operate directly on the byte encoding of text avoid these limitations, but until now they have lagged behind subword-based models in performance. Here we introduce Bolmo, a family of fully open byte-level LLMs that approach the capabilities of subword-based systems. Using a two-stage conversion procedure, we transform existing subword-based models into byte-level models with minimal additional training. The resulting models outperform prior byte-level approaches and excel on character-level reasoning tasks, while remaining competitive across standard benchmarks. By efficiently processing byte-level information, these models achieve practical inference speeds and can be adapted at low cost using the existing ecosystem around the source LLM. Our results remove a long-standing performance barrier to end-to-end byte-level language modeling, demonstrating that models operating on raw text encodings can scale competitively while offering advantages in domains requiring fine-grained textual understanding.

💡 Research Summary

The paper “Bolmo: Byteifying the Next Generation of Language Models” tackles a fundamental limitation of contemporary large language models (LLMs): they rely on subword tokenization, which compresses text into word pieces and consequently loses fine‑grained character information. This loss is especially detrimental for domains where meaning is encoded at the character level, such as source code, DNA sequences, or other scientific data. Moreover, subword tokenization introduces biases (e.g., dependence on token boundaries), restricts vocabulary flexibility, and forces a uniform compute allocation per token, which can be inefficient for long byte sequences.

To overcome these issues, the authors propose Bolmo, a family of fully open‑source byte‑level LLMs that achieve performance comparable to state‑of‑the‑art subword models while preserving character‑level fidelity. The key contribution is a two‑stage conversion pipeline that “byteifies” an existing subword model with less than 1 % of the original pre‑training budget (≈39.3 B tokens).

Stage 1 – Subword‑to‑Byte Distillation.
The source subword model (Olmo‑3 B/7 B or Olmo‑2 B/1 B) serves as a teacher. A byte‑level student model is initialized with the same architecture but operates on UTF‑8 bytes. The student is trained to exactly reproduce the teacher’s token‑level predictions by aligning byte‑level boundaries with subword boundaries. A specialized loss penalizes mismatches between the student’s predicted patch boundaries and the teacher’s subword splits, ensuring that the byte model learns the same linguistic behavior as the original.

Stage 2 – End‑to‑End Fine‑Tuning.
After the distillation phase, the entire pipeline (byte embedding → local encoder → global transformer → byte decoder) is fine‑tuned on the original data distribution. This step allows the model to exploit the richer granularity of byte inputs and to adjust any residual discrepancies introduced by the latent tokenization process.

Architecture – Latent Tokenizer Language Model (LTLM).
Bolmo follows the LTLM paradigm: raw bytes are first pooled into “byte patches” (e.g., groups of 4–8 bytes). Each patch is embedded via a large, sparsely‑activated embedding table. Crucially, the authors retain the original subword embeddings as a residual added to each byte embedding, effectively giving the model access to both byte‑level and subword‑level information. This hybrid embedding strategy dramatically improves the performance‑efficiency trade‑off without inflating inference latency. The pooled patch embeddings are processed by a deep transformer (the “global model”), after which a decoder de‑pools the representations back to the original byte sequence.

Experimental Evaluation.
The authors instantiate Bolmo‑7 B (derived from Olmo‑3 B/7 B) and Bolmo‑1 B (derived from Olmo‑2 B/1 B). Evaluation spans four axes:

1. Standard Benchmarks (MMLU, BIG‑Bench, etc.) – Bolmo matches or slightly exceeds the subword baselines, demonstrating that byte‑level processing does not sacrifice general language understanding.
2. Character‑Level Reasoning (CUTE, CodeEval, etc.) – Bolmo shows a striking absolute gain of +16.5 percentage points over the best prior public byte‑level model (BL‑T 7 B) and improves over the source Olmo model on tasks that require precise character manipulation.
3. Domain‑Specific STEM/Code/Biology Tasks – Bolmo outperforms Olmo on coding benchmarks and on biological sequence prediction, confirming the advantage of preserving per‑character semantics.
4. Efficiency – By increasing the bytes‑per‑patch ratio, Bolmo reduces the effective sequence length, yielding up to 1.4× faster inference with comparable memory usage. This scaling flexibility is unavailable to subword models, whose token granularity is fixed.

Ablation Studies.

• Non‑causal Patch Boundary Prediction: Removing the explicit boundary predictor speeds up inference but incurs a modest 1–2 pp performance drop, indicating the predictor’s role in fine‑grained accuracy.
• Necessity of Stage 1: Skipping the distillation stage leads to unstable training and a 5–7 pp degradation, confirming that the teacher‑student alignment is essential for rapid convergence.
• Local Encoder Design: Varying the size and sparsity of the local encoder shows a clear trade‑off; larger, sparser encoders improve performance with minimal latency impact.

Task Arithmetic Adaptation.
Bolmo can be “post‑trained” for specific domains without any additional gradient updates by leveraging the same task‑arithmetic techniques used for subword models. This enables rapid specialization (e.g., Python code generation) at essentially zero extra cost, a practical advantage for research labs with limited compute.

Conclusions and Future Work.
Bolmo demonstrates that byte‑level LLMs can close the performance gap with subword models while offering superior character‑level reasoning, better compute‑efficiency, and reduced language bias. The two‑stage conversion pipeline makes it feasible to upgrade existing open‑source subword models into byte‑level counterparts without the prohibitive expense of training from scratch. Future directions include scaling the approach to >30 B parameters, extending to multilingual UTF‑8 scripts (where byte patterns differ), and co‑designing hardware accelerators that natively support variable‑size byte patches for even faster inference.

In sum, the paper provides a compelling blueprint for the next generation of language models that operate directly on raw text encodings, unlocking new possibilities for scientific, technical, and multilingual applications.