用Mamba替代Transformer，发现其在图文生成上表现更好但视觉定位能力弱。
Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling
- 用结构化状态空间模型Mamba替换VLM中的Transformer
- Mamba在图像描述、问答任务上超越Transformer，但在视觉定位上差距扩大
- 适合需要全局摘要的任务，不适合需精确检索的场景
本研究探索将视觉语言模型（VLMs）中的Transformer替换为最近提出的结构化状态空间模型（SSM）Mamba。在控制条件下测试了最大30亿参数的模型，结果显示基于Mamba的VLM在图像描述、问答和阅读理解任务上优于基于Transformer的模型。然而，在视觉定位任务中，Transformer表现更优，且性能差距随规模增大而扩大。我们提出两个假设：1）任务无关的视觉编码对隐藏状态更新的影响；2）从上下文中进行多模态检索的难度。结果表明，任务感知编码对定位任务提升有限，而Transformer在上下文多模态检索中显著优于Mamba。总体而言，Mamba在依赖图像整体摘要的任务中表现良好，但在需要从上下文精确检索显式信息时表现不佳。
原文摘要 · Abstract (English)
This study explores replacing Transformers in Visual Language Models (VLMs) with Mamba, a recent structured state space model (SSM) that demonstrates promising performance in sequence modeling. We test models up to 3B parameters under controlled conditions, showing that Mamba-based VLMs outperforms Transformers-based VLMs in captioning, question answering, and reading comprehension. However, we find that Transformers achieve greater performance in visual grounding and the performance gap widens with scale. We explore two hypotheses to explain this phenomenon: 1) the effect of task-agnostic visual encoding on the updates of the hidden states, and 2) the difficulty in performing visual grounding from the perspective of in-context multimodal retrieval. Our results indicate that a task-aware encoding yields minimal performance gains on grounding, however, Transformers significantly outperform Mamba at in-context multimodal retrieval. Overall, Mamba shows promising performance on tasks where the correct output relies on a summary of the image but struggles when retrieval of explicit information from the context is required.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。