arXiv:2508.07375cs.CLcs.SD2025-08

通过动态分段对话轮次,实现语音与文本的精准交替生成。

TurnGuide: Enhancing Meaningful Full Duplex Spoken Interactions via Dynamic Turn-Level Text-Speech Interleaving

  • 动态划分对话轮次,实现文本与语音的逐轮交替生成
  • 在保持自然语音流的同时提升语义连贯性
  • 适合需要真实交互体验的语音助手与对话系统

全双工语音语言模型(FD-SLMs)旨在通过建模复杂的对话轮次行为(如打断、回应词和重叠说话)实现自然实时的语音交互。端到端(e2e)FD-SLMs利用真实双通道对话数据捕捉多说话人互动模式,但其对话能力常因长语音序列和高质量语音对话数据有限而弱于纯文本对话。尽管交错式文本-语音生成可缓解此问题,但将离散文本标记嵌入连续双通道音频流可能破坏流畅交互所需的时间对齐。为此,我们提出TurnGuide,一种新型的端到端FD-SLM文本-语音交错生成方法:动态将助手语音划分为对话轮次,并在轮次级别实现文本与语音的交替生成。该方法使FD-SLMs在不牺牲自然声学流的前提下融入大语言模型的语义智能。大量实验表明,TurnGuide显著提升了e2e FD-SLMs生成语义合理、连贯语音的能力,并在多种轮次事件上达到当前最优表现。演示地址:https://dreamtheater123.github.io/TurnGuide-Demo/。代码开源:https://github.com/dreamtheater123/TurnGuide。

原文摘要 · Abstract (English)

Full-Duplex Speech Language Models (FD-SLMs) are specialized foundation models designed to enable natural, real-time spoken interactions by modeling complex conversational turn-taking such as interruptions, backchannels, and overlapping speech. End-to-end (e2e) FD-SLMs leverage real-world double-channel conversational data to capture nuanced two-speaker dialogue patterns for human-like interactions, but their conversational abilities often degrade compared to pure-text conversation due to prolonged speech sequences and limited high-quality spoken dialogue data. Although interleaved text-speech generation could mitigate this degradation, integrating discrete text tokens into continuous double-channel audio streams could disrupt the precise time alignment required for fluid interaction. To address this, we propose TurnGuide, a novel text-speech interleaved generation approach for e2e FD-SLMs that dynamically segments assistant speech into dialogue turns and interleaves turn-level text and speech generation. This approach allows FD-SLMs to integrate the semantic intelligence of LLMs without compromising the natural acoustic flow. Extensive experiments show that TurnGuide not only significantly improves e2e FD-SLMs to produce semantically meaningful, coherent speech but also achieves state-of-the-art performance on various turn-taking events. Demos are available at https://dreamtheater123.github.io/TurnGuide-Demo/. Code is available at https://github.com/dreamtheater123/TurnGuide.

语音交互对话系统文本语音融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。