用浅层特征补足深层信息，提升视觉语言理解细节
MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understanding
- 用深层语义特征动态提取浅层细节，实现跨层特征融合
- 在LLaVA-1.5上显著提升多模态任务表现，推理更轻量
- 适合需要高精度视觉理解的场景，如细粒度图像问答
尽管多模态大语言模型在通过跨模态交互理解复杂人类意图方面取得显著进展，但捕捉图像细节仍具挑战。以往通过集成多个视觉编码器增强视觉细节的方法引入冗余和计算开销。我们观察到，大多数多模态大模型仅使用视觉编码器的最后一层特征图进行视觉表征，忽略了浅层特征图中丰富的细粒度信息。为此，我们提出 modelname，一种简单而有效的多层特征融合方法，可高效整合视觉变换器（ViTs）的深层与浅层特征。具体而言，它利用语义对齐的深层特征作为查询，动态从浅层特征中提取缺失细节，从而在保持语义一致性的前提下，丰富表征的细粒度信息。该方法应用于LLaVA-1.5模型，在视觉表征和基准测试性能上均取得显著提升，相比多编码器集成方案更具灵活性和轻量化优势。代码与模型已开源至 https://github.com/yuecao0119/MMFuser。
原文摘要 · Abstract (English)
Despite significant advancements in Multimodal Large Language Models (MLLMs) for understanding complex human intentions through cross-modal interactions, capturing intricate image details remains challenging. Previous methods integrating multiple vision encoders to enhance visual detail introduce redundancy and computational overhead. We observe that most MLLMs utilize only the last-layer feature map of the vision encoder for visual representation, neglecting the rich fine-grained information in shallow feature maps. To address this issue, we propose \modelname, a simple yet effective multi-layer feature fuser that efficiently integrates deep and shallow features from Vision Transformers (ViTs). Specifically, it leverages semantically aligned deep features as queries to dynamically extract missing details from shallow features, thus preserving semantic alignment while enriching the representation with fine-grained information. Applied to the LLaVA-1.5 model, \modelname~achieves significant improvements in visual representation and benchmark performance, providing a more flexible and lightweight solution compared to multi-encoder ensemble methods. The code and model have been released at https://github.com/yuecao0119/MMFuser.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。