让大模型直接实时语音对话，延迟低于100毫秒
IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities
- 用GroupFormer架构压缩语音序列，缩小语音与文本长度差距
- 在50万轮语音对话数据上训练，多轮对话延迟低于100ms
- 适合需要低延迟语音交互的智能助手、车载系统等场景
现有大模型语音交互方法依赖显式文本自回归生成以保证内容质量，但导致计算开销大、多轮交互延迟高。为此，我们提出IntrinsicVoice，一种具备内在实时语音交互能力的大模型。通过减少文本与语音之间的模态差距，其创新架构GroupFormer可将语音序列压缩至接近文本长度，显著缩短序列差异，加快推理速度，缓解长文本建模问题。同时，构建了包含近50万轮语音对语音对话的多轮语音对话数据集 extit{IntrinsicVoice-500k}，并设计跨模态训练策略以增强语音与文本的语义对齐。实验表明，IntrinsicVoice在多轮对话场景中能以低于100ms的延迟生成高质量语音响应。演示地址：https://instrinsicvoice.github.io/
原文摘要 · Abstract (English)
Current methods of building LLMs with voice interaction capabilities rely heavily on explicit text autoregressive generation before or during speech response generation to maintain content quality, which unfortunately brings computational overhead and increases latency in multi-turn interactions. To address this, we introduce IntrinsicVoic,e an LLM designed with intrinsic real-time voice interaction capabilities. IntrinsicVoice aims to facilitate the transfer of textual capabilities of pre-trained LLMs to the speech modality by mitigating the modality gap between text and speech. Our novelty architecture, GroupFormer, can reduce speech sequences to lengths comparable to text sequences while generating high-quality audio, significantly reducing the length difference between speech and text, speeding up inference, and alleviating long-text modeling issues. Additionally, we construct a multi-turn speech-to-speech dialogue dataset named \method-500k which includes nearly 500k turns of speech-to-speech dialogues, and a cross-modality training strategy to enhance the semantic alignment between speech and text. Experimental results demonstrate that IntrinsicVoice can generate high-quality speech response with latency lower than 100ms in multi-turn dialogue scenarios. Demos are available at https://instrinsicvoice.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。