发现视觉语言模型深层无需持续处理图像令牌,性能不降反升。
Do Vision Language Models Need to Process Image Tokens?
- 图像特征在浅层快速收敛,后续层变化极小
- 单标签任务可大幅减少视觉深度,多标签仍需完整视觉信息
- 图像令牌影响推理路径而非最终答案,适合研究模型内部机制者
视觉语言模型(VLMs)通过将视觉编码器与大语言模型结合取得显著成功。尽管其在深层Transformer中持续处理密集图像令牌(带来巨大计算开销),但这一过程是否必要仍不明确。本文系统研究发现,视觉表征在早期即快速收敛至有限复杂度状态:熵稳定、内在维度压缩、轨迹曲率趋于恒定;而文本表征则持续重构。一旦稳定,各层视觉表征基本可互换,深层变换有限。深度截断实验表明,单令牌任务对视觉深度削减较鲁棒,而多令牌生成仍需完整视觉输入。确定性解码下,降低视觉深度更显著扰动中间推理路径,说明图像令牌主要影响推理结构而非最终结论。这些发现挑战了‘深层视觉处理普遍必要’的默认假设,动摇现有多模态大模型架构基础。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) have achieved remarkable success by integrating visual encoders with large language models (LLMs). While VLMs process dense image tokens across deep transformer stacks (incurring substantial computational overhead), it remains fundamentally unclear whether sustained image-token processing is necessary for their performance or visual representations meaningfully evolve from early to later layers. In this work, we systematically investigate the functional role of image tokens in VLMs and show that visual representations rapidly converge to a bounded-complexity regime, \ie their entropy stabilizes, intrinsic dimensionality compresses, and trajectory curvature approaches a near-constant profile. In contrast, textual representations continue to undergo substantial restructuring across depth. Once stabilized, visual representations become largely interchangeable between layers, indicating limited additional transformation in deeper stages. Further, depth-wise visual truncation reveals that the necessity of visual processing is task-dependent, where single-token predictions remain comparatively robust to truncated visual depth, but multi-token generation require sustained access to visual representations. Under deterministic decoding, reducing visual depth perturbs intermediate reasoning trajectories more strongly than final outputs, suggesting that image tokens influence the structure of reasoning more than the ultimate conclusions. Collectively, these findings \textbf{question the assumption} that deeper visual processing is uniformly essential in VLMs, challenging the current paradigm of multimodal LLM architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。