提出听思说框架,实现语音交互中边说边思考,降低延迟。
LTS-VoiceAgent: A Listen-Think-Speak Framework for Efficient Streaming Voice Interaction via Semantic Triggering and Incremental Reasoning
- 用动态语义触发检测有意义语音前缀,决定何时开始思考。
- 双角色流调度器并行运行思考与说话,减少等待时间。
- 在多个数据集上优于传统串行方案,适合高实时性语音应用。
实时语音代理面临困境:端到端模型缺乏深度推理,而串行流水线因严格按语音识别(ASR)、大语言模型(LLM)推理、文本转语音(TTS)顺序执行导致高延迟,与人类对话中听众常在说话者未结束时就开始思考不同。尽管串行架构仍是复杂任务主流,现有流式策略通过固定分块或基于语音活动检测(VAD)的机械分割,或推测生成来降低延迟,但常破坏语义单元或浪费计算在需回滚的预测上。为此,我们提出LTS-VoiceAgent,一种听-思-说框架,明确分离‘何时思考’与‘如何增量推理’。其包含动态语义触发器以检测有意义的语音前缀,并采用双角色流调度器协调后台思考者(维持状态)与前台说话者(推测求解)。该并行设计实现‘边说边思考’而不阻塞响应。我们还引入一个含自然口误的暂停与修复基准测试(Pause-and-Repair),用于压力测试流式鲁棒性。在VERA、Spoken-MQA、BigBenchAudio及自建基准上的实验表明,LTS-VoiceAgent在准确率-延迟-效率权衡上优于串行基线和现有流式策略。
原文摘要 · Abstract (English)
Real-time voice agents face a dilemma: end-to-end models often lack deep reasoning, while cascaded pipelines incur high latency by executing ASR, LLM reasoning, and TTS strictly in sequence, unlike human conversation where listeners often start thinking before the speaker finishes. Since cascaded architectures remain the dominant choice for complex tasks, existing cascaded streaming strategies attempt to reduce this latency via mechanical segmentation (e.g., fixed chunks, VAD-based splitting) or speculative generation, but they frequently either break semantic units or waste computation on predictions that must be rolled back. To address these challenges, we propose LTS-VoiceAgent, a Listen-Think-Speak framework that explicitly separates when to think from how to reason incrementally. It features a Dynamic Semantic Trigger to detect meaningful prefixes, and a Dual-Role Stream Orchestrator that coordinates a background Thinker (for state maintenance) and a foreground Speaker (for speculative solving). This parallel design enables "thinking while speaking" without blocking responses. We also introduce a Pause-and-Repair benchmark containing natural disfluencies to stress-test streaming robustness. Experiments across VERA, Spoken-MQA, BigBenchAudio, and our benchmark show that LTS-VoiceAgent achieves a stronger accuracy-latency-efficiency trade-off than serial cascaded baselines and existing streaming strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。