分治式多模态模型,理解生成双优。
Decoupled Vision-Language System for Multimodal Understanding and Generation

- 视觉与语言系统分离,通过动态路由模块实现跨模态交互
- 在图文理解与文本生图任务上均超越现有模型表现
- 适合需要兼顾理解与生成能力的研究者和开发者
我们提出一种新型多模态大语言模型架构 Libra,支持多模态理解与生成。该架构包含独立的视觉系统与语言系统,通过跨模态桥接连接。设计上解耦自模态建模与跨模态交互,使各模态可学习独特表征,同时保持高效跨模态理解。解耦主要通过开关注意力模块与开关前馈网络模块实现,动态路由计算流以适应不同场景。我们在两个关键设置下评估:Libra-1 用于仅理解的图像到文本任务;Libra-2 用于统一的图像到文本理解与文本到图像生成任务。此外,我们还优化了标记化、位置编码与监督策略。实验表明,专用的 Libra 设计实现了理解与生成能力的相互促进,在多项理解与生成基准上均取得优异性能。
原文摘要 · Abstract (English)
We introduce a new architecture design for multimodal large language models (MLLMs), Libra, capable of both multimodal understanding and generation. Libra architecture contains one vision system and one language system, connected by cross-modal bridges. This design decouples self-modal modeling and cross-modal interaction, enabling each modality to learn its unique representations while maintaining effective cross-modal comprehension. The decoupling is mainly achieved in a switch attention module and a switch FFN module, which dynamically routes the computation flow for self-modal modeling and cross-modal interaction scenarios. We evaluate the effectiveness in two important settings: \textbf{Libra-1} for the understanding-only image-to-text setting, and \textbf{Libra-2} for unified image-to-text understanding and text-to-image generation. In addition to the architecture design, we discuss various improvements on tokenization, positional encoding, and supervision. Experiments demonstrate that the dedicated Libra design enables mutual improvements on multimodal understanding and generation, achieving strong performance on both understanding and generation benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。