提出空间感知投影器，用少75%视觉标记实现更好图文理解。
Spatial-Aware Efficient Projector for MLLMs via Multi-Layer Feature Aggregation
- 多层特征融合+改进深度可分离卷积，增强视觉标记空间信息。
- 视觉标记数减少75%，在多个基准上表现最优。
- 适合追求高效高精度图文理解的多模态模型研究者。
投影器在多模态语言模型（MLLMs）中起关键作用。其输出的视觉标记数量影响模型效率，而视觉标记质量则决定模型的视觉理解能力。当前投影器研究主要聚焦于减少视觉标记数量以提升效率，常忽视序列化二维视觉标记与自然语言标记之间的固有空间差异。为此，本文提出空间感知高效投影器（SAEP），通过在多层视觉特征上使用改进的可分离深度卷积模块，增强视觉标记的空间信息。结果表明，SAEP可将视觉标记数量减少75%，同时显著提升多模态空间理解能力。相较于现有投影器，SAEP在多个大规模多模态评估基准上表现最佳，验证了其在弥合模态差距方面的有效性。
原文摘要 · Abstract (English)
The projector plays a crucial role in multi-modal language models (MLLMs). The number of visual tokens it outputs affects the efficiency of the MLLM, while the quality of the visual tokens influences the visual understanding capabilities of the MLLM. Current explorations on the projector focus on reducing the number of visual tokens to improve efficiency, often overlooking the inherent spatial discrepancy between the serialized 2-dimensional visual token sequences and natural language token sequences. A Spatial-Aware Efficient Projector (SAEP) is proposed to address this issue. In detail, our SAEP method employs an modified separable depthwise convolution module on multi-layer visual features to enhance the spatial information of visual tokens. As a result, our SAEP method can not only largely reduce the number of visual tokens by 75\%, but also significantly improve the multimodal spatial understanding capability of MLLMs. Moreover, compared to existing projectors, our SAEP gets best performances on massive multimodal evaluation benchmarks, which denotes its effectiveness on bridging the modality gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。