让大模型同时听讲和回应,无需外部模块即可实现自然对话。
BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM

- 用特殊标记让单个自回归模型自主判断何时说话、听或停。
- 仅用40万条样本微调后,在评估中实现92%对话成功与100%打断成功率。
- 性能媲美甚至超越传统分步模型,适合实时语音交互应用。
实时全双工语音交互是下一代语音聊天机器人的关键特征,允许模型同时听和说,并处理重叠、犹豫和插话等自然现象。现有语音语言模型(SpeechLMs)如LLaMA-Omni和GLM-4-Voice仍为轮次式,依赖外部语音活动检测(VAD)模块标记用户发言结束,从根本上限制了交互能力。本文提出BayLing-Duplex,一种原生全双工SpeechLM,仅用一个自回归大模型自主决定何时听、何时说、何时停止,无需辅助转交模块。设计仅在标准词汇表中添加少量特殊标记,可跨模型迁移,复用现有训练与服务栈,无需架构改造。从公开的GLM-4-Voice检查点出发,仅使用40万条全双工样本微调并经过轻量DPO阶段,BayLing-Duplex在InstructS2S-Eval上达到92%的对话成功与100%打断成功率,语音响应评分从2.17提升至3.39(对比Moshi)。同时,在Llama Questions、Web Questions和Alpaca-Eval上表现不逊于其轮次式对应模型,证明同时听与说建模不会牺牲回答质量。
原文摘要 · Abstract (English)
Real-time, full-duplex speech interaction is a key feature of next-generation spoken chatbots, allowing the model to listen and speak at the same time and to handle natural phenomena such as overlap, hesitation, and barge-in. Existing speech language models (SpeechLMs) such as LLaMA-Omni and GLM-4-Voice are still turn-based and rely on an external Voice Activity Detection (VAD) module to mark the end of the user's turn, which fundamentally limits their interactive ability. In this paper, we introduce BayLing-Duplex, a native full-duplex SpeechLM where a single autoregressive LLM decides when to listen, when to speak, and when to stop, with no auxiliary turn-taking module. The design adds only a few special tokens to the standard vocabulary, so it transfers across LLMs and reuses existing training and serving stacks with no architectural adaptation. Starting from the public GLM-4-Voice checkpoint and using only 400K full-duplex samples for fine-tuning followed by a lightweight DPO stage, BayLing-Duplex reaches 92% turn-taking success and 100% interruption success on InstructS2S-Eval, while improving the speech-response score from 2.17 to 3.39 over Moshi. BayLing-Duplex also matches or surpasses its turn-based counterpart on Llama Questions, Web Questions, and Alpaca-Eval, showing that simultaneous listen-and-speak modeling does not sacrifice response quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。