arXiv:2604.05971cs.CVcs.CL2026-04

CLIP模型偏爱图像中心,边缘物体易被忽略,影响精细视觉理解。

Is CLIP Cross-Eyed? Revealing and Mitigating Center Bias in the CLIP Family

  • 通过嵌入分解与注意力分析,发现模型在聚合时丢失边缘信息
  • 中心区域的视觉特征主导最终表示,导致边缘物体识别失败
  • 无需训练的提示和注意力重分配可有效缓解该偏差

近期研究发现,对比视觉-语言模型如CLIP常缺乏对视觉内容的细粒度理解。我们识别出CLIP家族中一种持续存在的新型失效模式——中心偏差,即模型过度关注图像中心区域,忽视边界处的重要物体。这一缺陷从根本上限制了复杂任务的执行能力。通过从表征和注意力两个角度分析,利用嵌入分解与注意力图分析,我们发现相关概念(尤其是远离中心的物体)在最终表示中因视觉嵌入聚合过程中的信息损失而消失,尤其与池化机制密切相关。最后,我们证明可通过无需训练的策略(如视觉提示和注意力重分配)将模型注意力引导至非中心区域,有效缓解此偏差。

原文摘要 · Abstract (English)

Recent research has shown that contrastive vision-language models such as CLIP often lack fine-grained understanding of visual content. While a growing body of work has sought to address this limitation, we identify a distinct failure mode in the CLIP family, which we term center bias, that persists even in recent model variants. Specifically, CLIP tends to disproportionately focus on the central region of an image, overlooking important objects located near the boundaries. This limitation is fundamental as failure to recognize relevant objects makes it difficult to perform any sophisticated tasks that depend on those objects. To understand the underlying causes of the limitation, we conduct analyses from both representation and attention perspectives. Using interpretability methods, i.e., embedding decomposition and attention map analysis, we find that relevant concepts especially those associated with off-center objects vanish from the model's embedding in the final representation due to information loss during the aggregation of visual embeddings, particularly the reliance on pooling mechanisms. Finally, we show that this bias can be alleviated with training-free strategies such as visual prompting and attention redistribution by redirecting models' attention to off-center regions.

CLIP视觉偏差注意力机制模型解释

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。