用多标记预测加速语音生成，质量速度双提升
VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generation
- 采用多标记预测替代传统单标记预测，实现并行生成
- 1B/8B模型在有限数据下性能媲美主流全模态大模型
- 开源全部代码数据，支持实时语音交互研究
语音大语言模型（Speech LLM）已成为语音处理领域的研究热点。我们提出VocalNet-1B和VocalNet-8B，一系列高性能、低延迟的语音LLM，基于可扩展且模型无关的训练框架，支持实时语音交互。核心创新在于首次将多标记预测（MTP）应用于语音LLM，实现了从标准下一个标记预测（NTP）的范式转变，显著提升生成速度与质量。通过分析MTP对语音生成的影响并实验验证，设计了一种简单高效的MTP实现方案。实验表明，VocalNet在训练数据有限的情况下，性能可与主流全模态大模型相当，并显著超越现有开源语音LLM。为促进可复现性与社区发展，所有模型权重、推理代码、训练数据及框架实现均已公开于https://github.com/SJTU-OmniAgent/VocalNet。
原文摘要 · Abstract (English)
Speech large language models (LLMs) have emerged as a prominent research focus in speech processing. We introduce VocalNet-1B and VocalNet-8B, a series of high-performance, low-latency speech LLMs enabled by a scalable and model-agnostic training framework designed for real-time voice interaction. Central to our contribution is the first application of multi-token prediction (MTP) to speech LLMs. This approach represents a paradigm shift from standard next-token prediction (NTP), offering simultaneous improvements in generation speed and quality. Informed by analysis of MTP's effect on speech generation and experimental comparisons, we designed a straightforward and highly effective MTP implementation. Experiments demonstrate that VocalNet performs on par with mainstream Omni LLMs even with limited training data, and significantly surpasses existing open-source speech LLMs. To foster reproducibility and community advancement, all model weights, inference code, training data, and framework implementations have been made publicly available at https://github.com/SJTU-OmniAgent/VocalNet
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。