首个支持语音驱动的全流程任务代理，可自动调用工具完成复杂对话任务。
AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks
- 构建语音原生的多轮对话系统，通过模块化设计动态调用日历、搜索等工具。
- 在VoiceBench上达92.75%准确率，接近GPT-4o水平，人类评测任务成功率达90%。
- 开源可扩展，适合语音助手、智能客服等需自主决策的应用场景。
尽管语言与语音技术取得进展，目前仍无开源系统能实现端到端的语音到语音、多轮对话，并集成工具使用与代理推理。我们提出AURA（Agent for Understanding, Reasoning, and Automated Tool Use），首个开源的语音原生助手，可通过动态工具调用和多轮对话完成复杂目标驱动任务。AURA采用级联式开放权重语音识别（ASR）、文本转语音（TTS）与大模型（LLM）架构，支持日历预订、联系人查询、网络搜索和邮件发送等工具。其模块化设计允许仅通过自然语言提示和动作类定义即可快速集成新工具。在VoiceBench上，AURA在OpenBookQA任务中得分92.75%，优于所有开源权重系统，接近GPT-4o表现；在AlpacaEval上得分为4.39，与同类开源系统相当。人工评估显示，在复杂多轮语音任务中任务成功率达90%。
原文摘要 · Abstract (English)
Despite advances in language and speech technologies, no open-source system enables full speech-to-speech, multi-turn dialogue with integrated tool use and agentic reasoning. We introduce AURA (Agent for Understanding, Reasoning, and Automated Tool Use), the first open-source, speech-native assistant capable of completing complex, goal-driven tasks through dynamic tool invocation and multi-turn conversation. AURA combines open-weight ASR, TTS, and LLMs in a cascaded pipeline and supports tools such as calendar booking, contact lookup, web search, and email. Its modular design allows easy integration of new tools using natural language prompts and action classes. On VoiceBench, AURA scores 92.75% on OpenBookQA-outperforming all open-weight systems and nearing GPT-4o-and 4.39 on AlpacaEval, competitive with other open-weight systems. Human evaluation shows 90% task success on complex, multi-turn speech tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。