让音视频大模型测试时自动推理，不训练也能提升表现
Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMs
- 用演员-评论家框架在测试时提炼分步推理能力
- 在4500题基准上实现最高100%的性能提升
- 适合研究多模态推理或构建音视频AI系统的开发者
近期推理优化显著提升了大语言模型性能，但现有方法未充分应对音视频场景的复杂性。本文提出AURELIA，一种基于演员-评论家架构的音视频（AV）推理框架，可在测试时将结构化、分步推理过程蒸馏进音视频大模型（AVLLMs），无需额外训练或微调即可增强其处理复杂多模态输入的能力。为推动AVLLM推理能力发展，我们构建了包含4500个音视频问题的挑战性基准AVReasonBench，每个问题配有详细分步推理过程，涵盖六类任务，包括融合地理与文化知识的AV-GeoIQ任务。在该基准上评估18个AVLLMs，发现其多模态推理能力存在显著局限。使用AURELIA后，性能最高提升100%，验证了推理增强数据生成的有效性。代码与数据将公开于：https://github.com/schowdhury671/aurelia。
原文摘要 · Abstract (English)
Recent advancements in reasoning optimization have greatly enhanced the performance of large language models (LLMs). However, existing work fails to address the complexities of audio-visual scenarios, underscoring the need for further research. In this paper, we introduce AURELIA, a novel actor-critic based audio-visual (AV) reasoning framework that distills structured, step-by-step reasoning into AVLLMs at test time, improving their ability to process complex multi-modal inputs without additional training or fine-tuning. To further advance AVLLM reasoning skills, we present AVReasonBench, a challenging benchmark comprising 4500 audio-visual questions, each paired with detailed step-by-step reasoning. Our benchmark spans six distinct tasks, including AV-GeoIQ, which evaluates AV reasoning combined with geographical and cultural knowledge. Evaluating 18 AVLLMs on AVReasonBench reveals significant limitations in their multi-modal reasoning capabilities. Using AURELIA, we achieve up to a 100% relative improvement, demonstrating its effectiveness. This performance gain highlights the potential of reasoning-enhanced data generation for advancing AVLLMs in real-world applications. Our code and data will be publicly released at: https: //github.com/schowdhury671/aurelia.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。