让语音模型像人一样反复听音,提升复杂音频理解能力
Echo: Towards Advanced Audio Comprehension via Audio-Interleaved Reasoning
- 将音频作为主动推理组件,支持动态重听与持续感知
- 在专家级和通用任务上均超越现有模型,显著提升理解准确率
- 适合需要深度音频分析的研究者与开发者
大型音频语言模型(LALMs)的发展使人们对其复杂音频理解能力寄予厚望。当前方法多沿用文本推理范式,通过一次性编码处理音频内容,造成关键信息瓶颈。受人类认知启发,我们提出音频交错推理机制,将音频视为主动推理元素,实现持续音频参与和基于感知的分析。为此,我们设计两阶段训练框架:首先通过监督微调教会模型定位关键音频片段,再利用强化学习激励模型进行高效重听。同时构建结构化数据生成流程以生成高质量训练数据。最终提出 Echo 模型,可在推理过程中按需动态重听音频。在多个音频理解基准测试中,Echo 在挑战性专家级任务和通用任务上均表现优异。全面分析验证了该方法的高效性与泛化能力,确立其为推进音频理解的重要方向。项目页面:https://github.com/wdqqdw/Echo。
原文摘要 · Abstract (English)
The maturation of Large Audio Language Models (LALMs) has raised growing expectations for them to comprehend complex audio much like humans. Current efforts primarily replicate text-based reasoning by contextualizing audio content through a one-time encoding, which introduces a critical information bottleneck. Drawing inspiration from human cognition, we propose audio-interleaved reasoning to break through this bottleneck. It treats audio as an active reasoning component, enabling sustained audio engagement and perception-grounded analysis. To instantiate it, we introduce a two-stage training framework, first teaching LALMs to localize salient audio segments through supervised fine-tuning, and then incentivizing proficient re-listening via reinforcement learning. In parallel, a structured data generation pipeline is developed to produce high-quality training data. Consequently, we present Echo, a LALM capable of dynamically re-listening to audio in demand during reasoning. On audio comprehension benchmarks, Echo achieves overall superiority in both challenging expert-level and general-purpose tasks. Comprehensive analysis further confirms the efficiency and generalizability of audio-interleaved reasoning, establishing it as a promising direction for advancing audio comprehension. Project page: https://github.com/wdqqdw/Echo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。