用查询式投影压缩视觉信息，提升Mamba模型多模态效率
Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM
- 通过跨模态注意力动态压缩视觉令牌，减少输入长度
- 在多个视觉语言任务中提升模型性能与处理速度
- 无需人工设计图像特征的二维扫描顺序，简化流程
Transformer 模型随输入长度增长呈现二次复杂度，给大语言模型带来不可持续的计算负担。相比之下，选择性扫描结构化状态空间模型（Mamba）有效缓解了这一问题。本文提出一种基于查询的跨模态投影器，通过跨注意力机制根据输入压缩视觉令牌，从而增强 Mamba 在视觉-语言建模中的效率。该方法还消除了将原始图像特征转换为 Mamba 输入序列时需手动设计 2D 扫描顺序的必要。在多个视觉-语言理解基准测试中，实验结果表明，所提出的跨模态投影器显著提升了基于 Mamba 的多模态大模型的性能与吞吐量。
原文摘要 · Abstract (English)
The Transformer's quadratic complexity with input length imposes an unsustainable computational load on large language models (LLMs). In contrast, the Selective Scan Structured State-Space Model, or Mamba, addresses this computational challenge effectively. This paper explores a query-based cross-modal projector designed to bolster Mamba's efficiency for vision-language modeling by compressing visual tokens based on input through the cross-attention mechanism. This innovative projector also removes the need for manually designing the 2D scan order of original image features when converting them into an input sequence for Mamba LLM. Experimental results across various vision-language understanding benchmarks show that the proposed cross-modal projector enhances Mamba-based multimodal LLMs, boosting both performance and throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。