Beyond the Utterance: An Empirical Study of Very Long Context Speech Recognition

Automatic speech recognition (ASR) models are normally trained to operate over single utterances, with a short duration of less than 30 seconds. This choice has been made in part due to computational constraints, but also reflects a common, but often inaccurate, modelling assumption that treats utterances as independent and identically distributed samples. When long-format audio recordings are available, to work with such systems, these recordings must first be segmented into short utterances and processed independently. In this work, we show that due to recent algorithmic and hardware advances, this is no longer necessary, and current attention-based approaches can be used to train ASR systems that operate on sequences of over an hour in length. Therefore, to gain a better understanding of the relationship between the training/evaluation sequence length and performance, we train ASR models on large-scale data using 10 different sequence lengths from 10 seconds up to 1 hour. The results show a benefit from using up to 21.8 minutes of context, with up to a 14.2% relative improvement from a short context baseline in our primary experiments. Through modifying various architectural components, we find that the method of encoding positional information and the model’s width/depth are important factors when working with long sequences. Finally, a series of evaluations using synthetic data are constructed to help analyse the model’s use of context. From these results, it is clear that both linguistic and acoustic aspects of the distant context are being used by the model.

💡 Research Summary

The paper investigates how modern attention‑based speech recognizers can be trained and evaluated on very long audio sequences, up to one hour, without the need to segment recordings into short utterances. The authors combine three recent advances—(1) Flash Attention, an efficient GPU kernel that computes self‑attention without materialising the full quadratic‑size similarity matrix, (2) a FastConformer encoder that uses 8× subsampling and depth‑wise convolutions to reduce sequence length, and (3) a curriculum‑style sequence‑length warm‑up that starts training with short clips and gradually doubles the length until a predefined maximum. Together these techniques allow a Conformer‑CTC model to fit a full hour of audio into the memory of a single 180 GB A/H100 GPU.

Ten different maximum sequence lengths (10 s, 30 s, 1 min, 5 min, 10 min, 15 min, 20 min, 21.8 min, 30 min, 1 h) are explored on a massive dataset of 58 k hours of Spotify podcasts and the Floras‑50 conversational corpus. The authors also vary architectural factors such as the number of Conformer layers, model width (256 vs. 512 channels), and positional encoding schemes. Four positional encodings are compared: (i) no explicit encoding (relying on convolutional modules), (ii) sinusoidal, (iii) standard rotary (θ = 10 k), and (iv) high‑frequency rotary (θ = 1.5 M). The high‑frequency rotary encoding consistently yields the lowest word error rate (WER), indicating that a larger rotation frequency reduces the bias toward nearby frames and helps the model attend to distant context.

A central methodological contribution is the careful handling of “context fragmentation.” When long recordings are split into short windows, the start and end of each window lack past/future context, artificially inflating error rates for short‑sequence models. To avoid this confound, three evaluation schemes are introduced:

1. Moving‑Averaged Window – overlapping windows are processed and post‑softmax predictions are averaged; stride is set as a fraction of the window length.
2. Buffered Window – a buffer larger than the model’s receptive field is processed, but only the central region (which has full left and right context) contributes to the final output.
3. Sliding Window Attention (SWA) – the entire recording is fed to the model, but attention is restricted to a local window equal to the training sequence length, implemented with block‑sparse kernels.

These schemes ensure that any performance gain stems from genuine use of distant information rather than merely reduced fragmentation.

The experimental results reveal several key findings:

• Optimal context length: Performance improves with longer context up to about 21.8 minutes, where the relative WER reduction reaches 14.2 % compared with the 10‑second baseline. Beyond this point, gains plateau or slightly decline, suggesting diminishing returns for ultra‑long sequences.
• Architectural impact: Increasing model depth or width improves absolute WER but does not change the relative benefit of longer context. Positional encoding is the most influential factor; high‑frequency rotary encodings enable the model to exploit the full temporal span.
• Training dynamics: Directly training on long sequences (>20 s) leads to instability; the proposed warm‑up schedule is essential for convergence.
• Domain shift: When the test domain differs strongly from training (e.g., podcasts vs. meeting recordings), long‑context models achieve larger improvements. In‑domain evaluations show little benefit beyond 20–82 seconds.
• Noise robustness: Adding synthetic background noise (SNR = 0 dB) degrades short‑context models more severely; long‑context models retain a smaller WER increase, indicating that distant acoustic cues help disambiguate noisy segments.
• Synthetic analysis: Controlled experiments with artificially inserted linguistic cues demonstrate that the model uses both language‑level information (e.g., predicting a word based on preceding discourse) and acoustic information (e.g., speaker identity, background texture). However, linguistic exploitation appears limited to shallow n‑gram‑like dependencies rather than deep semantic understanding.

Overall, the study provides a comprehensive empirical picture of how modern ASR systems can be scaled to handle hour‑long recordings, what architectural choices matter for long‑range dependency modeling, and under which conditions such long context yields tangible benefits. The authors release code, checkpoints, and detailed results, paving the way for future work on streaming scenarios, memory‑efficient attention variants, and richer multimodal context integration.