发现视觉语言模型理解图中关系时，边信息出现较晚，影响推理能力。
Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Models
- 通过合成图数据探查模型内部表示，发现节点信息早于边信息被编码。
- 边信息在视觉编码器中无法线性分离，仅在文本令牌中线性表达。
- 该延迟现象解释了模型难以理解箭头方向等抽象关系的原因。
大型视觉语言模型（LVLMs）在图示理解任务上表现强劲，但仍难以理解元素间的关联，尤其是由节点和有向边（如箭头、线条）表示的关系。为探究这一局限的根本原因，我们基于有向图构建了精心设计的合成图数据集，对LVLMs的内部表征进行探测。实验表明，边信息在视觉编码器中不可线性分离，仅在语言模型的文本标记中才实现线性编码；而节点信息与全局结构特征已在视觉编码器的单个隐藏状态中实现线性编码。结果表明，不同类型的视觉信息形成线性可分表征的时间阶段不同。特别是边信息的延迟出现，可能解释了为何LVLMs在理解边方向等需要更高阶组合处理的任务上表现不佳。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) demonstrate strong performance on diagram understanding benchmarks, yet they still struggle with understanding relationships between elements, particularly those represented by nodes and directed edges (e.g., arrows and lines). To investigate the underlying causes of this limitation, we probe the internal representation of LVLMs using a carefully constructed synthetic diagram dataset based on directed graphs. Our probing experiments reveal that edge information is not linearly separable in the vision encoder and becomes linearly encoded only in the text tokens in the language model. In contrast, node information and global structural features are already linearly encoded in individual hidden states of the vision encoder. These findings suggest that the stage at which linearly separable representations are formed varies depending on the type of visual information. In particular, the delayed emergence of edge representations may help explain why LVLMs struggle with relational understanding, such as interpreting edge directions, which require more abstract, compositionally integrated processes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。