发现视觉语言模型中存在可操控的否定方向，无需微调即可提升对否定的理解。
When Negation Is a Geometry Problem in Vision-Language Models
- 在CLIP嵌入空间中找到否定相关方向，通过测试时干预实现否定感知。
- 新评估框架用多模态大模型判断图像内容，更可靠反映真实理解能力。
- 无需额外数据或训练，在罕见样本上仍具泛化能力，适合研究者参考。
联合视觉-语言嵌入模型如CLIP通常无法理解文本查询中的否定，例如在查询“一件无标识的蓝色纯色衬衫”中识别‘无’。以往工作主要通过大规模合成否定数据集进行微调来缓解该问题，但普遍采用基于检索的评估指标，难以真实反映否定理解能力。本文指出此类指标的两大局限，并提出基于多模态大模型作为裁判的新评估框架，其擅长回答关于图像内容的简单是非问题，能更公平地评估CLIP对否定的理解。我们进一步探究CLIP嵌入空间中是否存在与否定相关的方向，发现确有此类方向，并证明可通过测试时的表示工程干预，实现无需微调的否定感知行为。最后，我们在非常见图像-文本样本上测试否定理解能力，评估模型在分布外情况下的泛化性能。代码已公开于https://github.com/fawazsammani/negation-steering。
原文摘要 · Abstract (English)
Joint Vision-Language Embedding models such as CLIP typically fail at understanding negation in text queries, for example, failing to distinguish "no" in the query: "a plain blue shirt with no logos". Prior work has largely addressed this limitation through data-centric approaches, fine-tuning CLIP on large-scale synthetic negation datasets. However, these efforts are commonly evaluated using retrieval-based metrics that cannot reliably reflect whether negation is actually understood. In this paper, we identify two key limitations of such evaluation metrics and investigate an alternative evaluation framework based on Multimodal LLMs-as-a-judge, which typically excel at understanding simple yes/no questions about image content, providing a fair evaluation of negation understanding in CLIP models. We then ask whether there already exists a direction in the CLIP embedding space associated with negation. We find evidence that such a direction exists, and show that it can be manipulated through test-time intervention via representation engineering to steer CLIP toward negation-aware behavior without any fine-tuning. Finally, we test negation understanding on non-common image-text samples to evaluate generalization under distribution shifts. Code is at https://github.com/fawazsammani/negation-steering
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。