Distilling Token-Trained Models into Byte-Level Models

Byte Language Models (BLMs) have emerged as a promising direction for scaling language models beyond tokenization. However, existing BLMs typically require training from scratch on trillions of bytes, making them prohibitively expensive. In this paper, we propose an efficient distillation recipe that converts existing token-trained LLMs into BLMs while retaining comparable capabilities. Our recipe follows a two-stage curriculum: (1) Progressive Knowledge Distillation, which aligns byte-level representations with the embeddings of the token-trained teacher model; and (2) Byte-Level Supervised Fine-Tuning, which enables end-to-end generation entirely in the byte space. We validate our approach across multiple model families, including Llama, Qwen, and OLMo, and demonstrate that the distilled BLMs retain most of the teacher models’ performance using only approximately 125B bytes.

💡 Research Summary

The paper tackles the costly problem of training Byte‑Level Language Models (BLMs) from scratch by proposing a two‑stage distillation pipeline that converts existing token‑based Large Language Models (LLMs) into efficient BLMs. The authors observe that while BLMs eliminate tokenization bias and can handle any language uniformly, current approaches require trillions of bytes of data and massive compute, making them impractical for most research groups.

Stage 1 – Progressive Knowledge Distillation (PKD).
The first stage aligns the student byte model with the teacher token model through three sequential objectives:

1. Embedding Alignment (L_align). For each token boundary, the encoder output of the byte model at the corresponding byte position is forced to match the teacher’s static token embedding using an L2 loss. This creates a mapping from raw bytes to the semantic space learned by the token model.

2. Joint Distillation (L_distill). After synchronizing sequence lengths via the teacher’s token boundaries, the KL‑divergence between the teacher’s conditional token distribution and the student’s conditional byte‑derived distribution is minimized. This transfers higher‑level language knowledge while respecting the different granularities of the two models.

3. Boundary Learning (L_boundary). A novel “One‑Byte Lookahead Router” predicts chunk boundaries by comparing the hidden state of the current byte with that of the next byte. Binary cross‑entropy loss is applied using the teacher’s tokenizer boundaries as ground truth, enabling the student to learn an autonomous segmentation policy.

The three losses are applied sequentially (first L_align, then L_distill, finally L_boundary) rather than jointly, which stabilizes training and removes the need for extensive hyper‑parameter tuning.

Stage 2 – Byte‑Level Supervised Fine‑Tuning (SFT).
With the student model now able to produce token‑aligned representations, the second stage switches the entire pipeline to operate purely in the byte domain. Two strategies for boundary prediction during generation are explored:

• Joint Boundary Prediction (JBP). The output vocabulary is doubled (byte × 2) so that each token simultaneously encodes a byte value and a boundary flag. This allows the model to generate directly in an augmented space but increases vocabulary size.

• Multi‑Byte Prediction (MBP). An auxiliary head predicts the “next‑next” byte (x_{t+2}) while the primary head predicts the immediate next byte (x_{t+1}). The predicted embedding of x_{t+2} is fed into the router to compute the boundary probability for x_{t+1}, preserving causality while mimicking the look‑ahead behavior required by the router.

To preserve autoregressive properties during up‑sampling, the authors introduce Shifted‑Upsampling: chunk‑level representations are revealed only at the final byte of each chunk, while earlier bytes receive the representation of the preceding chunk (or a learned null token). This prevents future‑information leakage.

Experimental Validation.
The method is applied to three families of token‑based LLMs: Llama‑3.2‑3B, Qwen‑3‑4B, and OLMo‑1.5B. Using only ~125 B bytes of training data (≈1.25 × 10⁸ tokens), the distilled BLMs achieve 90‑96 % of the original models’ performance on a suite of benchmarks (MMLU, GSM‑8K, BBH, etc.). For example, on MMLU the distilled Llama‑3.2‑3B scores 51.8 versus the teacher’s 56.2, retaining over 92 % of the score. The authors also explore an On‑Policy Distillation step that further aligns the student’s policy with the teacher, yielding modest gains.

Ablation Studies show that each component of the curriculum is essential: removing embedding alignment drastically reduces early‑stage performance; omitting boundary learning leads to poor chunk segmentation; and using JBP versus MBP trades off memory usage against boundary accuracy.

Contributions are summarized as: (1) a novel two‑stage distillation framework that bridges token‑byte representation gaps; (2) the One‑Byte Lookahead Router and Shifted‑Upsampling mechanisms that ensure accurate boundary learning while preserving causality; (3) extensive cross‑model validation and analysis; (4) open‑sourcing of code, training scripts, and pretrained checkpoints.

Impact and Future Work.
By dramatically lowering the data and compute requirements for BLMs, this work makes byte‑level modeling accessible to a broader community and opens avenues for multilingual, multimodal, and low‑resource language research without reliance on handcrafted tokenizers. Future directions include scaling to larger models, integrating multimodal byte streams, and applying reinforcement‑learning‑based boundary policies.