用线性架构实现图文统一生成，训练数据少1000倍仍性能领先。
OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models
- 基于Mamba-2的线性结构，统一生成文本和图像。
- 仅用200万图文对训练，性能超Show-o，且推理快119倍。
- 适合追求高效多模态生成的开发者与研究者。
统一多模态理解与视觉生成模型因二次计算复杂度和依赖大规模训练数据而受限。本文提出OmniMamba，首个基于线性架构的多模态生成模型，通过统一的下一个词预测范式生成文本与图像。该模型充分借鉴Mamba-2的高计算与内存效率，将能力从文本扩展至多模态生成。为解决现有统一模型的数据低效问题，提出两项关键创新：(1) 解耦词汇表以引导模态特异性生成；(2) 任务专用LoRA实现参数高效适配。此外，引入解耦两阶段训练策略缓解双任务间数据不平衡。借助这些技术，OmniMamba在基准测试中表现媲美JanusFlow，超越Show-o，尽管仅使用200万图像-文本对进行训练，仅为Show-o的千分之一。值得注意的是，其推理效率显著，长序列生成相比基于Transformer的模型最高提速119.2倍，显存降低63%。代码与模型已开源于https://github.com/hustvl/OmniMamba。
原文摘要 · Abstract (English)
Recent advancements in unified multimodal understanding and visual generation (or multimodal generation) models have been hindered by their quadratic computational complexity and dependence on large-scale training data. We present OmniMamba, the first linear-architecture-based multimodal generation model that generates both text and images through a unified next-token prediction paradigm. The model fully leverages Mamba-2's high computational and memory efficiency, extending its capabilities from text generation to multimodal generation. To address the data inefficiency of existing unified models, we propose two key innovations: (1) decoupled vocabularies to guide modality-specific generation, and (2) task-specific LoRA for parameter-efficient adaptation. Furthermore, we introduce a decoupled two-stage training strategy to mitigate data imbalance between two tasks. Equipped with these techniques, OmniMamba achieves competitive performance with JanusFlow while surpassing Show-o across benchmarks, despite being trained on merely 2M image-text pairs, which is 1,000 times fewer than Show-o. Notably, OmniMamba stands out with outstanding inference efficiency, achieving up to a 119.2 times speedup and 63% GPU memory reduction for long-sequence generation compared to Transformer-based counterparts. Code and models are released at https://github.com/hustvl/OmniMamba
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。