DualTurn通过双通道语音预训练，让语音模型自然判断对话轮次。
DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining
- 双通道语音自回归生成，无标注学习对话动态
- 0.5B模型在动作预测上wF1达0.633，优于现有方法
- 适合需要精准轮次控制的语音助手研发
语音转语音模型能自然处理对话轮次，但难以支持工具调用或复杂推理；而现有的语音识别-大模型-语音合成（ASR-LLM-TTS）流水线依赖静音超时，导致轮次不自然。本文提出DualTurn，通过双通道对话音频的生成式预训练，让模型自回归生成两位说话人未来的语音，隐式学习对话动态，无需任何标签。随后在微调阶段，模型预测可解释的轮次信号，直接映射到代理动作。DualTurn持续监控双通道，提前预测轮次边界，生成五种代理动作。在标准基准测试中，0.5B的DualTurn在代理动作预测上的wF1达到0.633，显著优于VAP的0.389；在词级轮次预测上AUC为0.930，优于3.1B的音文模型（0.880），且更早预测轮次、减少打断。
原文摘要 · Abstract (English)
Speech-to-speech models handle turn-taking naturally but offer limited support for tool-calling or complex reasoning, while production ASR-LLM-TTS voice pipelines offer these capabilities but rely on silence timeouts, which lead to unnatural turn-taking. We present DualTurn, which narrows this gap through generative pretraining on dual-channel conversational audio. The model generates both speakers' future audio autoregressively, implicitly learning conversational dynamics without any labels, and is then fine-tuned to predict interpretable turn-taking signals that map directly to agent actions. DualTurn monitors both channels continuously, anticipating turn boundaries and producing five agent actions. On standard benchmarks, DualTurn (0.5B) outperforms both VAP on agent action prediction (wF1 0.633 vs. 0.389) and a 3.1B audio-text model on word-level turn prediction (AUC 0.930 vs. 0.880), while anticipating turn boundaries earlier with fewer interruptions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。