提出NTPP框架,让语音模型更好理解双通道对话的交互节奏。
NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction
- 用下一词对预测机制建模双通道语音对话
- 在换言预测和响应自然度上显著优于现有方法
- 推理延迟更低,适合实时语音交互场景
受GPT-4o强大能力启发,语音语言模型(SLMs)在实现自然流畅的人机语音交互方面日益受到关注。尽管已有多种SLM取得良好进展,但当前方法尚未充分挖掘双通道语音数据中蕴含的对话结构与动态特性。本文系统探索了现代大语言模型中双通道语音数据的应用,首次提出一种生成式建模范式——下一词对预测(Next-Token-Pair Prediction, NTPP),基于仅解码器架构实现无说话人依赖的双通道对话学习。我们在标准基准上评估该方法,实验证明NTPP在换言预测、响应连贯性与自然度方面显著提升SLMs的对话能力。此外,相比现有方法,NTPP实现显著更低的推理延迟,展现出在实时应用中的高效实用性。
原文摘要 · Abstract (English)
Inspired by the impressive capabilities of GPT-4o, there is growing interest in enabling speech language models (SLMs) to engage in natural, fluid spoken interactions with humans. Recent advancements have led to the development of several SLMs that demonstrate promising results in this area. However, current approaches have yet to fully exploit dual-channel speech data, which inherently captures the structure and dynamics of human conversation. In this work, we systematically explore the use of dual-channel speech data in the context of modern large language models, and introduce a novel generative modeling paradigm, Next-Token-Pair Prediction (NTPP), to enable speaker-independent dual-channel spoken dialogue learning using decoder-only architectures for the first time. We evaluate our approach on standard benchmarks, and empirical results show that our proposed method, NTPP, significantly improves the conversational abilities of SLMs in terms of turn-taking prediction, response coherence, and naturalness. Moreover, compared to existing methods, NTPP achieves substantially lower inference latency, highlighting its practical efficiency for real-time applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。