让语音对话模型像人一样思考和用工具,解决复杂任务
VoxMind: An End-to-End Agentic Spoken Dialogue System

- 引入'先思考再说话'机制,让模型规划推理更有序
- 任务完成率从34.88%提升至74.57%,超越Gemini-2.5-Pro
- 支持动态工具管理,大幅降低大工具集带来的延迟
近期端到端语音对话模型实现了自然交互。然而,随着用户需求日益复杂,仅依赖对话能力的模型常难以应对。引入代理能力至关重要:通过工具使用,模型可拓展知识边界并更好完成现实任务。但现有研究多聚焦感知与生成,对工具增强的探索有限。为此,我们提出VoxMind,一个集成框架,为端到端语音对话模型赋予全面代理能力。基于自建的470小时AgentChat数据集,我们引入'先思考再说话'机制,使模型将结构化推理作为规划与生成的前提。为缓解大规模工具集成带来的延迟瓶颈,提出多代理动态工具管理架构:通过辅助代理异步执行检索任务,与主模型推理轨迹对齐,有效解耦推理延迟与工具集规模。实验表明,VoxMind在代理性能上显著提升:任务完成率从34.88%增至74.57%,优于Gemini-2.5-Pro在语音代理任务的表现,同时保持通用对话质量。源代码与数据已公开于https://github.com/MM-Speech/VoxMind。
原文摘要 · Abstract (English)
Recent end-to-end spoken dialogue models enable natural interaction. However, as user demands become increasingly complex, models that rely solely on conversational abilities often struggle to cope. Incorporating agentic capabilities is therefore essential: by enabling tool use, these models can extend their knowledge boundaries and better solve real-world tasks. Yet, existing research has largely concentrated on core perception and generation, with comparatively limited exploration of such tool-augmented extensions. To bridge this gap, we present VoxMind, an integrated framework designed to equip end-to-end spoken dialogue models with comprehensive agentic abilities. Leveraging our curated 470-hour AgentChat dataset, we incorporate a "Think-before-Speak" mechanism, enabling the model to internalize structured reasoning as a critical prerequisite for planning and response generation. Furthermore, to mitigate latency bottlenecks caused by large-scale tool integration, we propose a Multi-Agent Dynamic Tool Management architecture. By asynchronously delegating retrieval tasks to an auxiliary agent aligned with the main model's reasoning trajectory, this system effectively decouples inference latency from toolset size. Experimental results confirm that VoxMind achieves significant improvements in agent performance: compared with strong baselines, the task completion rate increases from 34.88% to 74.57%, outperforming Gemini-2.5-Pro on spoken agent tasks while preserving general conversational quality. The source code and associated data are publicly available at https://github.com/MM-Speech/VoxMind.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。