让音频语言模型像工具一样协同工作，实现复杂问题的多步推理。
AudioToolAgent: An Agentic Framework for Audio-Language Models
- 用中央大模型调度音频工具，自动决定调用顺序和追问策略。
- 在MMAU等数据集上达到77.5%最高准确率，超越现有方法。
- 适合需要多轮语音理解与决策的智能助手、语音交互系统。
大型音频语言模型（LALMs）在音频理解任务中表现优异，但缺乏近期大语言模型（LLMs）所具备的多步推理与工具调用能力。本文提出AudioToolAgent框架，通过一个中央大语言模型代理协调多个音频语言模型作为工具，实现音频问答与语音转写。该代理无需直接访问音频，即可判断应调用哪些工具、如何生成后续问题，并仲裁不同工具输出的冲突。在MMAU、MMAR和MMAU-Pro数据集上的实验表明，该框架达到最优性能：在MMAU上达77.50%，在MMAR上达77.00%，在MMAU-Pro上达61.90%。基于Shapley值的分析揭示了高效的代理-工具组合。代码与复现材料已开源。
原文摘要 · Abstract (English)
Large Audio-Language Models (LALMs) perform well on audio understanding tasks but lack multistep reasoning and tool-calling found in recent Large Language Models (LLMs). This paper presents AudioToolAgent, a framework that coordinates audio-language models as tools via a central LLM agent that accesses tool adapters for audio question answering and speech-to-text. The agent reasons about which tools to invoke, how to formulate follow-up queries, and how to arbitrate conflicting tool outputs, without accessing the audio. Experiments with MMAU, MMAR, and MMAU-Pro show state-of-the-art accuracy: up to 77.50% in MMAU, 77.00% in MMAR, and 61.90% in MMAU-Pro. Shapley-based analysis identifies effective agent-tool combinations. The code and reproduction materials are available at https://github.com/GLJS/AudioToolAgent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。