Frame-Level Internal Tool Use for Temporal Grounding in Audio LMs

Large audio language models are increasingly used for complex audio understanding tasks, but they struggle with temporal tasks that require precise temporal grounding, such as word alignment and speaker diarization. The standard approach, where we generate timestamps as sequences of text tokens, is computationally expensive and prone to hallucination, especially when processing audio lengths outside the model’s training distribution. In this work, we propose frame-level internal tool use, a method that trains audio LMs to use their own internal audio representations to perform temporal grounding directly. We introduce a lightweight prediction mechanism trained via two objectives: a binary frame classifier and a novel inhomogeneous Poisson process (IHP) loss that models temporal event intensity. Across word localization, speaker diarization, and event localization tasks, our approach outperforms token-based baselines. Most notably, it achieves a >50x inference speedup and demonstrates robust length generalization, maintaining high accuracy on out-of-distribution audio durations where standard token-based models collapse completely.

💡 Research Summary

The paper tackles a fundamental limitation of current audio language models (ALMs) that generate timestamps as sequences of text tokens. While this token‑based approach is straightforward, it suffers from two major drawbacks: (1) computational inefficiency due to autoregressive decoding of multiple tokens for each timestamp, and (2) severe hallucination and length‑generalization failures when the model is asked to timestamp audio segments that lie outside the duration range seen during training. To address these issues, the authors propose “frame‑level internal tool use,” a method that trains an ALM to directly exploit its own internal frame‑wise representations for temporal grounding.

The core of the method is a lightweight prediction head attached to the decoder’s per‑frame outputs D = {d₁,…,d_T}. During training the head learns to map each frame to a probability that the target event (as described by a textual prompt) occurs at that frame. Two complementary loss functions are explored. The first is a binary cross‑entropy loss that treats each frame as an independent classification problem, with class‑reweighting to mitigate the extreme imbalance between event and non‑event frames. The second, and more novel, is an inhomogeneous Poisson process (IHP) loss. Here the model predicts a non‑negative intensity λ(t) for every frame by projecting d_t through a linear head and exponentiating the result. The total intensity Λ(T) = ∑ₜλ_t serves as a normalizing constant. For a set of n ground‑truth timestamps {t₁,…,t_n}, the negative log‑likelihood is –∑_{i=1}^n log λ(t_i) + n log Λ(T). This formulation encourages the model to concentrate probability mass on true event frames while suppressing spurious mass elsewhere, and it naturally handles multiple events and fuzzy annotations.

Inference proceeds by first obtaining the intensity curve λ(t) and its cumulative hazard Λ(t). Using the time‑rescaling theorem, the authors transform the inhomogeneous process into a homogeneous Poisson process, which yields a closed‑form posterior mode expressed as a product of a Beta‑distribution term (derived from the order statistics of event times) and the intensity λ(t). Because λ(t) is piecewise constant, the mode can be found efficiently by evaluating the log‑posterior at frame boundaries, resulting in a single parallel pass over the audio frames.

The approach is evaluated on three temporally demanding tasks: (i) word localization on LibriSpeech (using forced‑aligned word boundaries), (ii) speaker diarization on a multi‑speaker subset of VoxCeleb, and (iii) audio event localization on ESC‑50. Across all benchmarks, the frame‑level models outperform token‑based baselines in both mean absolute error and recall metrics. The IHP loss consistently beats the binary loss, especially when events are overlapping or annotations are imprecise. Speed measurements show a >50× inference acceleration because the model no longer generates multiple tokens per timestamp; instead it produces a probability vector for all frames in one forward pass. Moreover, when evaluated on out‑of‑distribution audio lengths (e.g., five‑minute recordings while trained only on ≤30‑second clips), the token‑based models collapse, whereas the frame‑level models maintain high accuracy, demonstrating robust length generalization.

The paper also discusses limitations. The current implementation uses a 25 Hz frame rate (40 ms frames), which may be insufficient for applications requiring sub‑10 ms precision. The IHP formulation assumes a single scalar intensity per frame, which could struggle with heavily overlapping events such as simultaneous speakers. Future work is suggested to incorporate multi‑intensity modeling, higher‑resolution frame encodings, and extensions to multimodal settings (e.g., video‑audio‑text) where similar internal‑tool concepts could be applied.

In summary, “frame‑level internal tool use” provides a principled, efficient, and more reliable way for audio language models to perform temporal grounding. By leveraging the model’s own internal representations rather than unconstrained text generation, it achieves dramatic speedups, eliminates hallucinated timestamps, and generalizes far beyond the training duration distribution—offering a compelling new direction for fine‑grained audio understanding systems.