arXiv:2605.20414eess.AS2026-05ACL

通过规划检索提升长音频理解,让大模型更准更快处理超长音频。

PlanRAG-Audio: Planning and Retrieval Augmented Generation for Long-form Audio Understanding

论文配图:PlanRAG-Audio: Planning and Retrieval Augmented Generation for Long-form Audio Understanding
图 1 · 摘自论文原文
  • 先规划所需音视频模态和时间范围,再精准检索相关内容。
  • 在30分钟以上音频上推理准确率提升18.7%,性能随时长稳定不下降。
  • 适合需要跨时段、多模态分析的长音频任务,如会议记录、播客解析。

长音频理解对大型音频语言模型(LALMs)构成重大挑战,源于音频序列极长以及需对分布于时间中的异质声学线索(如语音内容、说话人身份、情感、声音事件)进行推理。为此,我们提出基于规划的检索增强生成框架 PlanRAG-Audio,用于可扩展的长音频理解。不同于直接让 LALM 处理完整录音,PlanRAG-Audio 显式规划特定查询所需的模态与时间跨度,并从结构化文本与音频数据库中仅检索相关片段。该检索规划使模型能有效处理复杂跨域音频查询,同时大幅降低输入至大语言模型的长度。在广泛语音/音频检索任务上的实验表明,PlanRAG-Audio 在音频时长增加时仍保持性能稳定,推理准确率提升18.7%。

原文摘要 · Abstract (English)

Long-form audio understanding poses significant challenges for large audio language models (LALMs) due to the extreme length of audio sequences and the need to reason over heterogeneous acoustic cues distributed over time, such as speech content, speaker identity, emotion, and sound events. To address these challenges, we propose \textbf{PlanRAG-Audio}, a planning-based retrieval-augmented generation framework for scalable long-form audio understanding. Rather than having audio LALMs process entire recordings directly, PlanRAG-Audio explicitly plans which modalities and temporal spans are required for a given query, and retrieves only query-relevant information from a structured text and audio database. This retrieval planning enables effective reasoning over complex, cross-domain audio queries while substantially reducing the input length passed to the large language models. Experiments across a wide range of speech/audio retrieval demonstrate that PlanRAG-Audio improves reasoning accuracy and stabilizes performance as audio duration increases by decoupling inference cost from raw audio length.

长音频理解检索增强规划推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。