AudioRAG: A Challenging Benchmark for Audio Reasoning and Information Retrieval

Due to recent advancements in Large Audio-Language Models (LALMs) that demonstrate remarkable performance across a range of sound-, speech- and music-related tasks, there is a growing interest in proposing benchmarks to assess these models. Existing benchmarks generally focus only on reasoning with internal knowledge, neglecting real-world scenarios that require external information grounding. To bridge this gap, we introduce AudioRAG, a novel benchmark designed to evaluate audio-based reasoning augmented by information retrieval in realistic web environments. This benchmark comprises both LLM-generated and manually curated question-answer pairs. Our evaluations reveal that even the state-of-the-art LALMs struggle to answer these questions. We therefore propose an agentic pipeline that integrates audio reasoning with retrieval-augmented generation, providing a stronger baseline for future research.

💡 Research Summary

AudioRAG introduces a novel benchmark that explicitly tests the ability of large audio‑language models (LALMs) to perform multi‑hop reasoning over audio content while grounding their answers in up‑to‑date external knowledge. The authors first point out that existing audio reasoning benchmarks focus solely on internal, parametric knowledge, ignoring the fact that real‑world user queries often require information that is not stored in the model’s parameters, leading to hallucinations. To fill this gap, they construct a dataset of 500 question‑answer pairs drawn from two sources. The first source leverages publicly available audio datasets (MMAU, CinePile, FMA, iNaturalist, etc.). For each audio clip they extract a salient attribute from the metadata (e.g., music genre, animal species, speech transcript) and prompt GPT‑4o to generate multi‑hop questions that implicitly require the model to infer the attribute from the audio and then retrieve additional facts from the web. The second source consists of audio tracks manually harvested from online videos; human annotators listen to these clips and write contemporary, time‑sensitive questions that are unlikely to be covered by the training data of current LALMs. Each question may be presented as text or as an audio prompt, while the answer is always provided in text form.

To ensure high quality, the authors apply a two‑stage filtering pipeline. First, a “question validity” filter, involving both LLM checks and human annotators, removes items that lack a unique correct answer. Second, an “answer correctness” filter uses an LLM equipped with a live search tool to re‑answer each question based on the provided ground‑truth audio attributes; any discrepancy triggers human review and possible discarding or revision of the item.

The benchmark is then used to evaluate six recent LALMs, including open‑source models (Qwen2.5‑Omni, Audio Flamingo 3, Audio‑Reasoner, Baichuan‑Omni, Qwen3‑Omni) and the closed‑source Gemini‑2.5‑Flash. Raw performance is modest: accuracies range from 20 % (Audio‑Reasoner) to 45 % (Gemini‑Flash), with the open‑source Qwen family consistently outperforming the other open models. These results confirm that current LALMs excel at perceptual audio tasks but struggle with complex reasoning chains and external knowledge retrieval.

To address these shortcomings, the authors propose an “agentic pipeline” that couples a text‑centric reasoning LLM with two specialized tools: (1) an audio‑processing module (Tₐ) that extracts required audio attributes when the LLM issues a query wrapped in … tags, and (2) a deep‑web explorer (Tₑₓₚ) that performs live web searches via the Google Search API. The pipeline follows a Think‑Call‑Answer loop: the reasoning LLM iteratively thinks about the current state, decides which tool to invoke, receives the tool’s output, and updates its reasoning trace. The final answer is generated after the LLM has integrated both audio‑derived evidence and retrieved textual facts.

In experiments, Qwen3‑8B serves as the reasoning LLM, while Qwen2.5‑Omni or Qwen3‑Omni act as the audio processing back‑ends. Both the reasoning LLM and the audio tool are served via vLLM on four A100 GPUs. The agentic pipeline yields substantial gains: Qwen2.5‑Omni’s accuracy improves from 32.2 % to 39.5 % (a 22.7 % relative increase) and Qwen3‑Omni’s from 37.0 % to 46.2 % (a 24.9 % relative increase). Error analysis categorises failures into reasoning errors, audio‑processing errors, knowledge errors, and invalid answers; the pipeline notably reduces knowledge‑related and reasoning errors.

The paper concludes that integrating retrieval‑augmented generation with audio‑specific processing is essential for next‑generation multimodal assistants. AudioRAG provides a publicly released, reproducible benchmark and a strong baseline, encouraging future work on larger datasets, richer toolsets, and tighter human‑agent collaboration to further close the gap between perceptual audio understanding and real‑world, knowledge‑grounded reasoning.