通过分析音频贡献度,提升大模型在语音问答中的真实音频理解能力。
Measuring Audio's Impact on Correctness: Audio-Contribution-Aware Post-Training of Large Audio Language Models
- 按音频贡献度划分数据,分阶段训练以增强模型对音频的依赖。
- 在多个评测集上达到78.2%以上准确率,刷新语音问答性能纪录。
- 适合研究多模态模型训练策略或语音理解任务的开发者参考。
大型音频语言模型(LALMs)是多模态AI的重要方向,涵盖多样化的音频任务。近期,后训练技术显著提升了基础模型性能。尽管单阶段强化学习(RL)表现良好,但监督微调(SFT)+RL的多阶段方法仍不理想。现有研究未充分探索多阶段训练中数据分配策略,且缺乏高质量大规模数据集。为此,我们提出AudioMCQ,一个包含57.1万样本的音频多选题数据集,配备两种链式思维标注。我们发现LALMs普遍存在零音频贡献现象,即模型仅凭文本作答而不处理音频。为此提出音频贡献过滤,将数据分为弱与强音频贡献子集。基于此,设计两种有效后训练范式:弱到强(先在弱音频贡献数据上SFT,再在强数据上RL)和混合到强(先在混合数据上SFT,再在强数据上RL)。利用AudioMCQ,我们在DCASE 2025语音问答挑战赛中获第一名。采用不同训练策略,在MMAU-test-mini、MMAU、MMAR和MMSU上分别取得78.2%、75.6%、67.1%和70.7%的准确率,建立新基准。
原文摘要 · Abstract (English)
Large Audio Language Models (LALMs) represent an important frontier in multimodal AI, addressing diverse audio tasks. Recently, post-training of LALMs has received increasing attention due to significant performance improvements over foundation models. While single-stage post-training such as reinforcement learning (RL) has demonstrated promising results, multi-stage approaches such as supervised fine-tuning (SFT) followed by RL remain suboptimal. The allocation of data across multiple training stages to maximize LALM capabilities has not been fully explored, and large-scale, high-quality datasets for such research are also lacking. To address these problems, we firstly present AudioMCQ, a comprehensive audio multiple-choice question dataset comprising 571k samples with two kinds of chain-of-thought annotations. Secondly, we investigate the prevalent zero audio-contribution phenomenon in LALMs, where models derive correct answers solely from textual information without processing audio content. We propose Audio-Contribution Filtering to partition data into weak and strong audio-contribution subsets. Based on these insights, we develop two effective post-training paradigms: Weak-to-Strong (SFT on weak audio-contribution data followed by RL on strong audio-contribution data) and Mixed-to-Strong (SFT on mixed audio-contribution data followed by RL on strong audio-contribution data). We achieve first place in the DCASE 2025 Audio-Question-Answering challenge by using AudioMCQ. Additionally, leveraging our dataset with different training strategies, we achieve 78.2\% on MMAU-test-mini, 75.6\% on MMAU, 67.1\% on MMAR, and 70.7\% on MMSU, establishing new state-of-the-art performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。