用结构化思维和课程引导强化学习,提升音频语言模型的推理能力。
SARI: Structured Audio Reasoning via Curriculum-Guided Reinforcement Learning
- 设计结构化思维链,通过课程式强化学习训练音频语言模型。
- 在MMAU-mini上达到67.08%准确率,比基线提升16.35%。
- 结构化思维和渐进难度课程显著增强模型泛化与收敛速度。
现有研究显示,强化学习可通过让大语言模型‘先思考再回答’显著提升其推理能力。但该方法在音频-语言推理任务中的效果仍不明确。本文将DeepSeek-R1的组相对策略优化(GRPO)框架拓展至大音频-语言模型(LALM),构建了包含32,000样本的多选题语料库。采用两阶段训练:先在结构化与非结构化思维链上进行监督微调,再通过课程引导的GRPO进行强化学习,系统对比了隐式与显式、结构化与自由形式推理在相同架构下的表现。所提出的结构化音频推理模型SARI(Structured Audio Reasoning via Curriculum-Guided Reinforcement Learning)在基线模型Qwen2-Audio-7B-Instruct上实现平均准确率提升16.35%。基于Qwen2.5-Omni的变体在MMAU test-mini基准上达到67.08%的最先进水平。消融实验表明:(i) SFT预热对稳定强化学习训练至关重要;(ii) 结构化思维链比非结构化更具鲁棒性;(iii) 由易到难的课程设置可加速收敛并提升最终性能。结果表明,显式结构化推理与课程学习能显著提升音频-语言理解能力。
原文摘要 · Abstract (English)
Recent work shows that reinforcement learning(RL) can markedly sharpen the reasoning ability of large language models (LLMs) by prompting them to "think before answering." Yet whether and how these gains transfer to audio-language reasoning remains largely unexplored. We extend the Group-Relative Policy Optimization (GRPO) framework from DeepSeek-R1 to a Large Audio-Language Model (LALM), and construct a 32k sample multiple-choice corpus. Using a two-stage regimen supervised fine-tuning on structured and unstructured chains-of-thought, followed by curriculum-guided GRPO, we systematically compare implicit vs. explicit, and structured vs. free form reasoning under identical architectures. Our structured audio reasoning model, SARI (Structured Audio Reasoning via Curriculum-Guided Reinforcement Learning), achieves a 16.35% improvement in average accuracy over the base model Qwen2-Audio-7B-Instruct. Furthermore, the variant built upon Qwen2.5-Omni reaches state-of-the-art performance of 67.08% on the MMAU test-mini benchmark. Ablation experiments show that on the base model we use: (i) SFT warm-up is important for stable RL training, (ii) structured chains yield more robust generalization than unstructured ones, and (iii) easy-to-hard curricula accelerate convergence and improve final performance. These findings demonstrate that explicit, structured reasoning and curriculum learning substantially enhances audio-language understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。