让语音对话像真人一样流畅，直接处理语音流不依赖文本
Real-Time Textless Dialogue Generation
- 直接输入语音流，端到端生成语音回应，跳过文本中间表示
- 响应延迟极低，支持自然轮换对话，减少机械感
- 融入语气、笑声等副语言信号，适合语音助手与实时交互场景
大型语言模型的进展推动了文本对话系统的显著提升，能够生成准确且连贯的高质量回复。然而，语音对话系统在自然度方面仍落后，常出现响应慢、回答泛化或保守、缺乏节奏感和流畅换话等问题。这主要源于传统级联设计中各组件独立顺序处理，以及依赖文本作为中间表示。本文提出一种实时、无文本的语音对话生成模型（RTTL-DG），通过直接处理流式语音对话，实现低延迟响应和自然的轮换对话。该模型还融合了附带信号（如附和语、笑声、过滤词等），这些在传统级联系统中通常缺失，从而增强交互的人性化。代码与生成样例已开源：https://github.com/mailong25/rts2s-dg
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) have led to significant progress in text-based dialogue systems. These systems can now generate high-quality responses that are accurate and coherent across a wide range of topics and tasks. However, spoken dialogue systems still lag behind in terms of naturalness. They tend to produce robotic interactions, with issues such as slow response times, overly generic or cautious replies, and a lack of natural rhythm and fluid turn-taking. This shortcoming is largely due to the over-reliance on the traditional cascaded design, which involve separate, sequential components, as well as the use of text as an intermediate representation. This paper propose a real-time, textless spoken dialogue generation model (RTTL-DG) that aims to overcome these challenges. Our system enables fluid turn-taking and generates responses with minimal delay by processing streaming spoken conversation directly. Additionally, our model incorporates backchannels, filters, laughter, and other paralinguistic signals, which are often absent in cascaded dialogue systems, to create more natural and human-like interactions. The implementations and generated samples are available in our repository: https://github.com/mailong25/rts2s-dg
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。