揭秘视觉语言模型如何看图：关键信息藏在查询词里
What's in the Image? A Deep-Dive into the Vision of Vision Language Models
- 用查询词（如'描述图像'）就能生成完整描述，无需看图像块
- 中间层（约25%层）主导跨模态信息流动，前后层作用小
- 物体细节直接从图像对应区域提取，空间定位精准
视觉语言模型（VLMs）在理解复杂视觉内容方面表现卓越，但其处理视觉信息的内在机制仍不清晰。本文通过实证分析各层注意力模块，揭示若干关键发现：(i) 查询词（如'描述图像'）的内部表示被用于存储全局图像信息，模型仅凭这些查询词即可生成高度描述性的回答，无需直接访问图像令牌；(ii) 跨模态信息流主要受中间层（约全部层数的25%）影响，早期和晚期层贡献微弱；(iii) 细粒度视觉属性与对象细节以空间局部化方式直接从图像令牌中提取，特定对象或属性对应的生成令牌强烈关注图像中对应区域。我们提出新的定量评估方法，基于真实复杂视觉场景验证上述观察。最后，展示了这些发现对提升当前先进VLMs高效视觉处理的潜力。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have recently demonstrated remarkable capabilities in comprehending complex visual content. However, the mechanisms underlying how VLMs process visual information remain largely unexplored. In this paper, we conduct a thorough empirical analysis, focusing on attention modules across layers. We reveal several key insights about how these models process visual data: (i) the internal representation of the query tokens (e.g., representations of "describe the image"), is utilized by VLMs to store global image information; we demonstrate that these models generate surprisingly descriptive responses solely from these tokens, without direct access to image tokens. (ii) Cross-modal information flow is predominantly influenced by the middle layers (approximately 25% of all layers), while early and late layers contribute only marginally.(iii) Fine-grained visual attributes and object details are directly extracted from image tokens in a spatially localized manner, i.e., the generated tokens associated with a specific object or attribute attend strongly to their corresponding regions in the image. We propose novel quantitative evaluation to validate our observations, leveraging real-world complex visual scenes. Finally, we demonstrate the potential of our findings in facilitating efficient visual processing in state-of-the-art VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。