综述多模态大模型如何统一理解生成各类模态信息
From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities
- 构建统一框架，将多种非语言模态映射到大模型嵌入空间
- 提出两阶段训练策略，支持任意模态组合的交互与生成
- 适合入门者快速掌握多模态大模型研究脉络
为应对真实场景中的复杂任务，越来越多研究聚焦于全能型多模态大模型（Omni-MLLMs），旨在实现对各类模态的统一理解与生成。此类模型突破特定非语言模态的限制，将多种非语言模态映射至大语言模型的嵌入空间，使单一模型可处理任意模态组合的交互与理解。本文系统梳理相关研究，提供关于 Omni-MLLMs 的全面综述。首先，提出四类核心组件的细致分类体系，为统一多模态建模提供新视角；其次，介绍基于两阶段训练的有效融合方法，并讨论对应数据集与评估方式；最后，总结当前 Omni-MLLMs 的主要挑战并展望未来方向。希望本工作能作为初学者入门指南，推动该领域发展。相关资源已公开于 https://github.com/threegold116/Awesome-Omni-MLLMs。
原文摘要 · Abstract (English)
To tackle complex tasks in real-world scenarios, more researchers are focusing on Omni-MLLMs, which aim to achieve omni-modal understanding and generation. Beyond the constraints of any specific non-linguistic modality, Omni-MLLMs map various non-linguistic modalities into the embedding space of LLMs and enable the interaction and understanding of arbitrary combinations of modalities within a single model. In this paper, we systematically investigate relevant research and provide a comprehensive survey of Omni-MLLMs. Specifically, we first explain the four core components of Omni-MLLMs for unified multi-modal modeling with a meticulous taxonomy that offers novel perspectives. Then, we introduce the effective integration achieved through two-stage training and discuss the corresponding datasets as well as evaluation. Furthermore, we summarize the main challenges of current Omni-MLLMs and outline future directions. We hope this paper serves as an introduction for beginners and promotes the advancement of related research. Resources have been made publicly available at https://github.com/threegold116/Awesome-Omni-MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。