揭示视觉大模型推理时图像与文本令牌的四阶段交互规律
Phase Diagram of Vision Large Language Models Inference: A Perspective from Interaction across Image and Instruction
- 通过分析隐藏状态向量，揭示多模态交互随深度演进的四个阶段
- 早期层实现跨模态对齐，后期层增强跨模态融合，晚期层准备输出
- 适用于研究多模态模型内部机制的学者，尤其关注视觉语言协同
视觉大语言模型（VLLMs）通常将图像标记嵌入与文本标记嵌入拼接后进行因果建模，但其内部行为仍不明确，特别是两类标记间的交互机制。为探究推理过程中这种多模态交互，本文测量了来自不同模态的标记在隐藏状态向量中的上下文化程度。实验发现，随着Transformer模型深度增加，VLLMs表现出四阶段推理动态：(I) 对齐阶段：极早层中，跨模态上下文化出现，表明特征空间对齐；(II) 模态内编码：早期层中，模态内上下文化增强，而跨模态交互被抑制，体现为模态内局部编码；(III) 模态间编码：后期层中，跨模态上下文化增强，表明深层模态融合；(IV) 输出准备：极晚层中，全局上下文化减少，隐藏状态趋向解嵌入空间。
原文摘要 · Abstract (English)
Vision Large Language Models (VLLMs) usually take input as a concatenation of image token embeddings and text token embeddings and conduct causal modeling. However, their internal behaviors remain underexplored, raising the question of interaction among two types of tokens. To investigate such multimodal interaction during model inference, in this paper, we measure the contextualization among the hidden state vectors of tokens from different modalities. Our experiments uncover a four-phase inference dynamics of VLLMs against the depth of Transformer-based LMs, including (I) Alignment: In very early layers, contextualization emerges between modalities, suggesting a feature space alignment. (II) Intra-modal Encoding: In early layers, intra-modal contextualization is enhanced while inter-modal interaction is suppressed, suggesting a local encoding within modalities. (III) Inter-modal Encoding: In later layers, contextualization across modalities is enhanced, suggesting a deeper fusion across modalities. (IV) Output Preparation: In very late layers, contextualization is reduced globally, and hidden states are aligned towards the unembedding space.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。