开源双模态对话轮次检测模型，提升全双工语音交互自然性
Easy Turn: Integrating Acoustic and Linguistic Modalities for Robust Turn-Taking in Full-Duplex Spoken Dialogue Systems
- 融合声学与语言双模态信息，实时判断对话状态
- 在1145小时数据集上达到当前最优轮次检测准确率
- 适合语音助手、智能客服等需要自然对话的场景
全双工交互对自然的人机对话至关重要，但其核心挑战在于鲁棒的轮次检测，即系统需判断何时发言、倾听或保持沉默。现有方案多依赖专用轮次检测模型，且多数未开源；少数开源模型或参数量过大，或仅支持单一模态（如声学或语言）。另有方法通过微调大语言模型实现全双工，但需大量全双工数据，而公开可用的数据仍稀缺。为此，我们提出 Easy Turn，一个开源、模块化的轮次检测模型，融合声学与语言双模态信息，可预测四种对话状态：完整、不完整、回应信号和等待。同时发布 Easy Turn trainset，一个包含1145小时语音的开源数据集，用于训练轮次检测模型。相比现有开源模型如 TEN Turn Detection 与 Smart Turn V2，我们的模型在开源 Easy Turn testset 上达到最先进准确率。模型与数据将公开于 GitHub。
原文摘要 · Abstract (English)
Full-duplex interaction is crucial for natural human-machine communication, yet remains challenging as it requires robust turn-taking detection to decide when the system should speak, listen, or remain silent. Existing solutions either rely on dedicated turn-taking models, most of which are not open-sourced. The few available ones are limited by their large parameter size or by supporting only a single modality, such as acoustic or linguistic. Alternatively, some approaches finetune LLM backbones to enable full-duplex capability, but this requires large amounts of full-duplex data, which remain scarce in open-source form. To address these issues, we propose Easy Turn, an open-source, modular turn-taking detection model that integrates acoustic and linguistic bimodal information to predict four dialogue turn states: complete, incomplete, backchannel, and wait, accompanied by the release of Easy Turn trainset, a 1,145-hour speech dataset designed for training turn-taking detection models. Compared to existing open-source models like TEN Turn Detection and Smart Turn V2, our model achieves state-of-the-art turn-taking detection accuracy on our open-source Easy Turn testset. The data and model will be made publicly available on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。