让多模态模型看更浅层视觉特征,能更好完成精细视觉任务。
Multimodal Language Models See Better When They Look Shallower
- 分析ViT各层特征,发现浅层更适配精细视觉理解
- 在60+任务中,浅/中层特征使计数、定位等任务性能提升
- 提出轻量融合方法,优于单一层或专用融合方案
多模态大语言模型(MLLMs)通常从预训练Vision Transformer(ViT)的深层提取视觉特征,这种深层偏见主要源于经验惯例而非理论分析。尽管先前研究指出,ViT的不同层级捕获不同信息:浅层关注细粒度视觉细节,深层更贴近文本语义,但其对MLLM性能的影响仍不明确。本文首次系统研究了MLLMs的视觉层选择问题,通过分析ViT各层表示相似性,划分出浅、中、深三层组。在涵盖60多个任务的10个基准上,对1.4B至7B参数的MLLM进行评估,发现深层在语义丰富任务如OCR中表现优异,而浅层和中层在计数、定位、目标检测等细粒度任务中显著优于深层。基于此,我们提出一种轻量级特征融合方法,有策略地引入浅层特征,在多项任务中持续超越单层与专用融合基线。本工作首次为MLLM的视觉层选择提供了原则性依据,表明模型往往‘看得更浅’时,反而‘看得更好’。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) typically extract visual features from the final layers of a pretrained Vision Transformer (ViT). This widespread deep-layer bias, however, is largely driven by empirical convention rather than principled analysis. While prior studies suggest that different ViT layers capture different types of information, with shallower layers focusing on fine visual details and deeper layers aligning more closely with textual semantics, the impact of this variation on MLLM performance remains underexplored. We present the first comprehensive study of visual layer selection for MLLMs, analyzing representation similarity across ViT layers to establish shallow, middle, and deep layer groupings. Through extensive evaluation of MLLMs (1.4B-7B parameters) across 10 benchmarks encompassing 60+ tasks, we find that while deep layers excel in semantic-rich tasks like OCR, shallow and middle layers significantly outperform them on fine-grained visual tasks including counting, positioning, and object localization. Building on these insights, we propose a lightweight feature fusion method that strategically incorporates shallower layers, achieving consistent improvements over both single-layer and specialized fusion baselines. Our work offers the first principled study of visual layer selection in MLLMs, showing that MLLMs can often see better when they look shallower.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。