揭示CLIP模型如何通过注意力机制学会左右关系判断
Left-Right Symmetry Breaking in CLIP-style Vision-Language Models Trained on Synthetic Spatial-Relation Data
- 用1D图像文本对测试模型对左右关系的理解能力
- 标签多样性比布局多样性更能促进模型泛化表现
- 发现位置与词元嵌入交互导致左右不对称注意力分布
空间理解仍是视觉语言模型的核心挑战。然而，这种理解是否真正习得，以及其机制仍不明确。本文构建了一个可控的1维图像-文本测试平台，探究基于Transformer的视觉与文本编码器在使用CLIP式对比目标训练时，如何涌现出左右关系理解能力。我们端到端训练轻量级Transformer编码器，使用单对象与双对象场景的配对描述，并在系统性变化标签与布局多样性条件下评估对未见物体对的泛化能力。结果表明，对比学习能有效习得左右关系；其中，标签多样性比布局多样性是泛化的主要驱动力。通过注意力分解分析，我们发现位置嵌入与词元嵌入之间的交互诱导出水平方向的注意力梯度，破坏了左右对称性；移除该贡献会显著降低左右辨别能力。研究为CLIP类模型如何获得关系认知提供了机制层面的洞察。
原文摘要 · Abstract (English)
Spatial understanding remains a key challenge in vision-language models. Yet it is still unclear whether such understanding is truly acquired, and if so, through what mechanisms. We present a controllable 1D image-text testbed to probe how left-right relational understanding emerges in Transformer-based vision and text encoders trained with a CLIP-style contrastive objective. We train lightweight Transformer-based vision and text encoders end-to-end on paired descriptions of one- and two-object scenes and evaluate generalization to unseen object pairs while systematically varying label and layout diversity. We find that contrastive training learns left-right relations and that label diversity, more than layout diversity, is the primary driver of generalization in this setting. To gain the mechanistic understanding, we perform an attention decomposition and show that interactions between positional and token embeddings induce a horizontal attention gradient that breaks left-right symmetry in the encoders; ablating this contribution substantially reduces left-right discrimination. Our results provide a mechanistic insight of when and how CLIP-style models acquire relational competence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。