开源大模型实现低延迟语音交互，直接听语音指令生成语音回复。
LLaMA-Omni: Seamless Speech Interaction with Large Language Models
- 用语音编码器+适配器+大模型+流式解码器构建端到端语音交互架构。
- 响应延迟仅226毫秒，20万条语音指令数据训练，内容与风格更优。
- 4块显卡3天内完成训练，适合快速部署语音交互应用。
GPT-4o等模型通过语音实现实时人机交互，显著提升体验，但针对开源大语言模型（LLM）的语音交互研究仍不足。为此，我们提出LLaMA-Omni，一种面向低延迟、高质量语音交互的新型模型架构。该模型集成预训练语音编码器、语音适配器、LLM及流式语音解码器，无需语音转文字，可直接根据语音指令同时生成文本与语音响应。基于最新Llama-3.1-8B-Instruct模型构建，并构建包含20万条语音指令与对应语音回复的InstructS2S-200K数据集以对齐语音交互场景。实验表明，相比以往语音-语言模型，LLaMA-Omni在内容与风格上表现更优，响应延迟低至226ms。此外，模型训练仅需4块GPU运行不到3天，为未来高效开发语音-语言模型提供可行路径。
原文摘要 · Abstract (English)
Models like GPT-4o enable real-time interaction with large language models (LLMs) through speech, significantly enhancing user experience compared to traditional text-based interaction. However, there is still a lack of exploration on how to build speech interaction models based on open-source LLMs. To address this, we propose LLaMA-Omni, a novel model architecture designed for low-latency and high-quality speech interaction with LLMs. LLaMA-Omni integrates a pretrained speech encoder, a speech adaptor, an LLM, and a streaming speech decoder. It eliminates the need for speech transcription, and can simultaneously generate text and speech responses directly from speech instructions with extremely low latency. We build our model based on the latest Llama-3.1-8B-Instruct model. To align the model with speech interaction scenarios, we construct a dataset named InstructS2S-200K, which includes 200K speech instructions and corresponding speech responses. Experimental results show that compared to previous speech-language models, LLaMA-Omni provides better responses in both content and style, with a response latency as low as 226ms. Additionally, training LLaMA-Omni takes less than 3 days on just 4 GPUs, paving the way for the efficient development of speech-language models in the future.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。