Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts

Hybrid Transformer architectures, which combine softmax attention blocks and recurrent neural networks (RNNs), have shown a desirable performance-throughput tradeoff for long-context modeling, but their adoption and studies are hindered by the prohibitive cost of large-scale pre-training from scratch. Some recent studies have shown that pre-trained softmax attention blocks can be converted into RNN blocks through parameter transfer and knowledge distillation. However, these transfer methods require substantial amounts of training data (more than 10B tokens), and the resulting hybrid models also exhibit poor long-context performance, which is the scenario where hybrid models enjoy significant inference speedups over Transformer-based models. In this paper, we present HALO (Hybrid Attention via Layer Optimization), a pipeline for distilling Transformer models into RNN-attention hybrid models. We then present HypeNet, a hybrid architecture with superior length generalization enabled by a novel position encoding scheme (named HyPE) and various architectural modifications. We convert the Qwen3 series into HypeNet using HALO, achieving performance comparable to the original Transformer models while enjoying superior long-context performance and efficiency. The conversion requires just 2.3B tokens, less than 0.01% of their pre-training data

💡 Research Summary

The paper addresses two major obstacles that have limited the adoption of hybrid Transformer‑RNN models for very long contexts: (1) the need for massive amounts of data (tens to hundreds of billions of tokens) to distill a pre‑trained Transformer into a hybrid architecture, and (2) the severe degradation of long‑context performance after conversion, which defeats the primary advantage of hybrids (higher inference speed on long sequences).
To solve these problems, the authors introduce HALO (Hybrid Attention via Layer Optimization), a three‑stage distillation pipeline combined with a principled attention‑layer selection strategy, and HypeNet, a hybrid architecture that incorporates a novel positional encoding scheme called HyPE.

HALO Pipeline

1. Initialization (Attention‑weight transfer) – Every softmax‑attention layer of the teacher Transformer is replaced by an RNN layer of the same dimensionality, re‑using the Q, K, V, O projection matrices. Any RNN‑specific parameters (e.g., transition matrix Fₜ) are initialized with empirically‑derived defaults.
2. Stage 1 – Hidden‑state alignment – Each instantiated RNN is trained independently to minimize the mean‑squared error between its hidden states and those produced by the original attention layer. This aligns the representations before the full model is assembled, preventing large performance drops later.
3. Attention‑layer selection – The authors propose a performance‑driven importance score: for each layer i, replace it with its RNN counterpart, evaluate the drop in recall (R) and commonsense‑reasoning (CSR) on a validation suite, and compute

\