让语音模型边说边想,回答更聪明还无延迟。
STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Models
- 交替生成默想内容和语音响应,实现思考与说话同步
- 数学推理任务上比基线提升15%,且无额外延迟
- 适合需要自然流畅语音交互的智能助手场景
语音语言模型(SLMs)接收语音输入并生成语音回复。然而,当前的SLMs缺乏在回应前进行内部默想的能力。人类在表达前通常会进行复杂思维,从而清晰简洁地传达想法。因此,将未发声的思考过程融入SLMs具有重要意义。虽然直接生成完整思维链(CoT)可实现思考,但会因思维链长度不可控而增加语音响应延迟。为此,我们提出Stitch:一种交替生成未发声推理片段与语音响应片段的新生成方法。由于语音片段的播放时长远大于生成该片段所需时间,模型可利用空闲时间生成下一阶段的默想内容。当一段语音播放给用户时,模型继续生成下一段默想,实现思考与说话同步。令人惊讶的是,Stitch在数学推理数据集上性能比无法生成默想的基线模型高15%,且延迟与基线相当;在非推理任务上表现相当。项目演示见:https://d223302.github.io/STITCH。
原文摘要 · Abstract (English)
Spoken Language Models (SLMs) are designed to take speech inputs and produce spoken responses. However, current SLMs lack the ability to perform an internal, unspoken thinking process before responding. In contrast, humans typically engage in complex mental reasoning internally, enabling them to communicate ideas clearly and concisely. Thus, integrating an unspoken thought process into SLMs is highly desirable. While naively generating a complete chain-of-thought (CoT) reasoning before starting to talk can enable thinking for SLMs, this induces additional latency for the speech response, as the CoT reasoning can be arbitrarily long. To solve this issue, we propose Stitch, a novel generation method that alternates between the generation of unspoken reasoning chunks and spoken response chunks. Since the audio duration of a chunk of spoken response is much longer than the time to generate the tokens in a chunk of spoken response, we use the remaining free time to generate the unspoken reasoning tokens. When a chunk of audio is played to the user, the model continues to generate the next unspoken reasoning chunk, achieving simultaneous thinking and talking. Remarkably, Stitch matches the latency of baselines that cannot generate unspoken CoT by design while outperforming those baselines by 15% on math reasoning datasets; Stitch also performs equally well on non-reasoning datasets as those baseline models. Some animations and demonstrations are on the project page: https://d223302.github.io/STITCH.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。