探索多层视觉特征融合，提升多模态大模型性能。
Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices
- 系统分析不同视觉层对模型的影响，找出最优特征层组合。
- 多层特征在输入阶段直接融合效果最好，且稳定性强。
- 适合研究多模态模型特征融合的开发者和研究人员参考。
近年来，多模态大语言模型（MLLMs）取得了显著进展，视觉特征在提升模型性能方面作用日益重要。然而，多层视觉特征在MLLM中的融合仍缺乏深入研究，尤其在最佳层级选择与融合策略方面。现有方法常依赖随意设计，导致效果不佳。本文系统研究了多层视觉特征融合的两个核心问题：（1）选择最有效的视觉层级；（2）确定与语言模型的最佳融合方式。实验表明，整合多个阶段的视觉特征可提升泛化能力，但同一阶段增加额外特征会降低性能。此外，多层视觉特征在输入阶段直接融合，在各种配置下均表现出更优且更稳定的性能。相关代码已开源：https://github.com/EIT-NLP/Layer_Select_Fuse_for_MLLM。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have made significant advancements in recent years, with visual features playing an increasingly critical role in enhancing model performance. However, the integration of multi-layer visual features in MLLMs remains underexplored, particularly with regard to optimal layer selection and fusion strategies. Existing methods often rely on arbitrary design choices, leading to suboptimal outcomes. In this paper, we systematically investigate two core aspects of multi-layer visual feature fusion: (1) selecting the most effective visual layers and (2) identifying the best fusion approach with the language model. Our experiments reveal that while combining visual features from multiple stages improves generalization, incorporating additional features from the same stage typically leads to diminished performance. Furthermore, we find that direct fusion of multi-layer visual features at the input stage consistently yields superior and more stable performance across various configurations. We make all our code publicly available: https://github.com/EIT-NLP/Layer_Select_Fuse_for_MLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。