arXiv:2602.23266cs.CL2026-02被引 1

让语音对话系统边听边想、边想边说,显著降低延迟。

Discourse-Aware Dual-Track Streaming Response for Low-Latency Spoken Dialogue Systems

  • 小模型生成连接词,大模型并行推理,实现听思同步
  • 三阶段流水线重叠运行,响应延迟降低19%-51%
  • 适配多种大模型,对长句也稳定有效

实现类人响应是级联语音对话系统的关键挑战。传统ASR-LLM-TTS流程严格串行,需完整转录和全部推理后才能合成语音,导致高延迟。本文提出话语感知双轨流式响应框架(DDTSR),支持听而思、说而思。其核心机制包括:(1) 连接词引导的小-大模型协同,小模型生成低承诺度话语连接词,大模型并行进行知识密集型推理;(2) 流式跨模态协作,动态重叠ASR、LLM推理与TTS,提前可发声时刻;(3) 基于课程学习的话语连贯性增强,保持早期回应与后续推理输出的一致性。在两个语音对话基准上的实验表明,DDTSR将响应延迟降低19%-51%,同时维持话语质量。进一步分析显示,DDTSR可作为即插即用模块,兼容多种LLM主干,在不同语句长度下仍具鲁棒性,具备强实用性与可扩展性。

原文摘要 · Abstract (English)

Achieving human-like responsiveness is a critical yet challenging goal for cascaded spoken dialogue systems. Conventional ASR-LLM-TTS pipelines follow a strictly sequential paradigm, requiring complete transcription and full reasoning before speech synthesis can begin, which results in high response latency. We propose the Discourse-Aware Dual-Track Streaming Response (DDTSR) framework, a low-latency architecture that enables listen-while-thinking and speak-while-thinking. DDTSR is built upon three key mechanisms: (1) connective-guided small-large model synergy, where an auxiliary small model generates minimal-committal discourse connectives while a large model performs knowledge-intensive reasoning in parallel; (2) streaming-based cross-modal collaboration, which dynamically overlaps ASR, LLM inference, and TTS to advance the earliest speakable moment; and (3) curriculum-learning-based discourse continuity enhancement, which maintains coherence and logical consistency between early responses and subsequent reasoning outputs. Experiments on two spoken dialogue benchmarks demonstrate that DDTSR reduces response latency by 19%-51% while preserving discourse quality. Further analysis shows that DDTSR functions as a plug-and-play module compatible with diverse LLM backbones, and remains robust across varying utterance lengths, indicating strong practicality and scalability for real-time spoken interaction.

语音对话低延迟流式处理大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。