首个无需语音预训练的实时双向语音对话模型，支持插话与低比特率输出。
SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model
- 通过通道融合直接建模用户与语音代理的同步流，实现连续输入与响应。
- 比特率降至0.6 kbps（减半），推理能力优于已有模型，支持实时插话。
- 首次开源完整代码，可快速适配任意大语言模型构建双向语音交互系统。
语音对话是人机交互的自然形式，但现有语音语言模型多局限于回合制交流，缺乏实时插话等适应性。本文提出一种新型双向语音到语音（S2S）架构，支持用户连续输入与编码器代理输出的并行建模，通过通道融合直接处理双流信号。利用预训练流式编码器处理用户输入，使该模型成为首个无需语音预训练的双向S2S模型。用户与代理分别建模的结构便于编码器微调以优化代理语音，并将比特率降低至0.6 kbps（较此前工作减半）。实验表明，该模型在推理、换轮与插话能力上均优于已有方法。由于跳过语音预训练，所需语音数据显著减少，极大简化了基于任意大语言模型构建双向语音系统的流程。最后，它是首个公开提供训练与推理代码的双向S2S模型，有助于研究复现。
原文摘要 · Abstract (English)
Spoken dialogue is an intuitive form of human-computer interaction, yet current speech language models often remain constrained to turn-based exchanges, lacking real-time adaptability such as user barge-in. We propose a novel duplex speech to speech (S2S) architecture featuring continuous user inputs and codec agent outputs with channel fusion that directly models simultaneous user and agent streams. Using a pretrained streaming encoder for user input enables the first duplex S2S model without requiring speech pretrain. Separate architectures for agent and user modeling facilitate codec fine-tuning for better agent voices and halve the bitrate (0.6 kbps) compared to previous works. Experimental results show that the proposed model outperforms previous duplex models in reasoning, turn-taking, and barge-in abilities. The model requires significantly less speech data, as speech pretrain is skipped, which markedly simplifies the process of building a duplex S2S model from any LLMs. Finally, it is the first openly available duplex S2S model with training and inference code to foster reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。