打造可插拔的全双工语音交互系统,支持自然实时对话。
FireRedChat: A Pluggable, Full-Duplex Voice Interaction System with Cascaded and Semi-Cascaded Implementations
- 设计可插拔的模块化架构,融合个性化语音活动检测与语义停顿判断。
- 半级联版本降低延迟、减少错误传播,提升响应连贯性与鲁棒性。
- 适用于需要真实对话体验的智能助手与客服系统开发。
全双工语音交互允许用户与智能体同时说话并实现可控打断,带来更自然的助手和客服体验。现有方案要么是端到端难以控制,要么是依赖非开源组件的模块化流水线,限制整体优化。本文提出一个完整的全双工语音交互系统,包含话轮控制器、交互模块与对话管理器。控制器集成流式个性化语音活动检测(pVAD),抑制噪声及非主讲人误打断,精准标注主讲人段落,并明确支持主讲人打断;语义停顿检测器提升停止判断准确性。该系统可升级半双工流水线为全双工,支持级联与半级联两种实现方式。半级联版本捕捉情感与副语言线索,生成更连贯回应,降低延迟与错误传播,增强鲁棒性。对话管理器通过工具调用与上下文管理扩展功能。我们提出三项系统级指标:打断准确率、停顿检测准确率与端到端延迟,用于评估自然度、控制精度与效率。实验显示,误打断减少,语义停顿判断更准,延迟接近工业级水平,实现稳定、自然、实时的全双工交互。演示见:https://fireredteam.github.io/demos/firered_chat。
原文摘要 · Abstract (English)
Full-duplex voice interaction allows users and agents to speak simultaneously with controllable barge-in, enabling lifelike assistants and customer service. Existing solutions are either end-to-end, difficult to design and hard to control, or modular pipelines governed by turn-taking controllers that ease upgrades and per-module optimization; however, prior modular frameworks depend on non-open components and external providers, limiting holistic optimization. In this work, we present a complete, practical full-duplex voice interaction system comprising a turn-taking controller, an interaction module, and a dialogue manager. The controller integrates streaming personalized VAD (pVAD) to suppress false barge-ins from noise and non-primary speakers, precisely timestamp primary-speaker segments, and explicitly enable primary-speaker barge-ins; a semantic end-of-turn detector improves stop decisions. It upgrades heterogeneous half-duplex pipelines, cascaded, semi-cascaded, and speech-to-speech, to full duplex. Using internal models, we implement cascaded and semi-cascaded variants; the semi-cascaded one captures emotional and paralinguistic cues, yields more coherent responses, lowers latency and error propagation, and improves robustness. A dialogue manager extends capabilities via tool invocation and context management. We also propose three system-level metrics, barge-in, end-of-turn detection accuracy, and end-to-end latency, to assess naturalness, control accuracy, and efficiency. Experiments show fewer false interruptions, more accurate semantic ends, and lower latency approaching industrial systems, enabling robust, natural, real-time full-duplex interaction. Demos: https://fireredteam.github.io/demos/firered_chat.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。