开源复现GPT-4o多模态能力，支持视觉听觉输入与实时语音响应
Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities
- 三阶段训练对齐多模态，用有限数据实现跨模态理解
- 集成预训练视觉与音频编码器，保持各模态性能不下降
- 支持命令式打断，实现更自然的双向交互，适合研究多模态交互
GPT-4o作为多模态大模型的里程碑，具备视觉、听觉与文本理解能力，可直接输出语音并支持灵活的双向交互。当前开源模型虽能实现部分功能如视觉理解或语音对话，但统一建模仍面临多模态数据复杂性、模型架构与训练流程等挑战。本文提出Mini-Omni2，一个能对视觉与音频输入实时生成端到端语音响应的多模态助手。通过引入预训练视觉与听觉编码器，模型在单模态任务中保持高性能。我们设计三阶段训练流程，使语言模型在小规模数据上完成多模态对齐，从而处理多模态输入与输出。为提升交互灵活性，引入基于命令的打断机制。据我们所知，Mini-Omni2是目前最接近GPT-4o功能的开源实现，有望为后续研究提供重要参考。
原文摘要 · Abstract (English)
GPT-4o, an all-encompassing model, represents a milestone in the development of large multi-modal language models. It can understand visual, auditory, and textual modalities, directly output audio, and support flexible duplex interaction. Models from the open-source community often achieve some functionalities of GPT-4o, such as visual understanding and voice chat. Nevertheless, training a unified model that incorporates all modalities is challenging due to the complexities of multi-modal data, intricate model architectures, and training processes. In this paper, we introduce Mini-Omni2, a visual-audio assistant capable of providing real-time, end-to-end voice responses to visoin and audio queries. By integrating pretrained visual and auditory encoders, Mini-Omni2 maintains performance in individual modalities. We propose a three-stage training process to align modalities, allowing the language model to handle multi-modal inputs and outputs after training on a limited dataset. For interaction, we introduce a command-based interruption mechanism, enabling more flexible interaction with users. To the best of our knowledge, Mini-Omni2 is one of the closest reproductions of GPT-4o, which have similar form of functionality, and we hope it can offer valuable insights for subsequent research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。