arXiv:2510.06917cs.CLeess.AS2025-10ACL被引 16

让语音模型边听边思考,实现实时互动与提前打断。

SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models

  • 语音分块流式输入,接收即生成隐式推理链
  • 在用户说话时即可识别错误并提前打断,准确率提升37.1%
  • 可提前完成56.9%的工具调用,适合高实时性对话场景

当前大语言模型和语音语言模型通常在用户说完后才开始思考与响应,导致交互延迟高,不适用于需要实时对话的语音任务。我们提出SHANKS框架,模拟人类‘边听边想’的自然行为,使语音语言模型在接收用户语音输入的同时,持续生成未发声的思维链(chain-of-thought)。该框架将输入语音按固定时长分块,每块到达后立即基于已接收语音和历史推理生成新思考,并据此决定是否打断用户或调用工具。实验表明,在用户逐步解数学题时,SHANKS的打断准确率比无思考基线高出37.1%;在工具增强对话中,56.9%的工具调用可在用户结束前完成。整体上,SHANKS推动模型实现全程持续思考,而非仅在回合结束后才响应。

原文摘要 · Abstract (English)

Current large language models (LLMs) and spoken language models (SLMs) begin thinking and taking actions only after the user has finished their turn. This prevents the model from interacting during the user's turn and can lead to high response latency while it waits to think. Consequently, thinking after receiving the full input is not suitable for speech-to-speech interaction, where real-time, low-latency exchange is important. We address this by noting that humans naturally "think while listening." In this paper, we propose SHANKS, a general inference framework that enables SLMs to generate unspoken chain-of-thought reasoning while listening to the user input. SHANKS streams the input speech in fixed-duration chunks and, as soon as a chunk is received, generates unspoken reasoning based on all previous speech and reasoning, while the user continues speaking. SHANKS uses this unspoken reasoning to decide whether to interrupt the user and to make tool calls to complete the task. We demonstrate that SHANKS enhances real-time user-SLM interaction in two scenarios: (1) when the user is presenting a step-by-step solution to a math problem, SHANKS can listen, reason, and interrupt when the user makes a mistake, achieving 37.1% higher interruption accuracy than a baseline that interrupts without thinking; and (2) in a tool-augmented dialogue, SHANKS can complete 56.9% of the tool calls before the user finishes their turn. Overall, SHANKS moves toward models that keep thinking throughout the conversation, not only after a turn ends. Animated illustrations of Shanks can be found at https://d223302.github.io/SHANKS/

语音交互实时推理思维链对话系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。