现有视觉语言模型几乎不懂否定，新基准测试发现其表现接近随机
Vision-Language Models Do Not Understand Negation
- 构建涵盖18种任务的NegBench基准，覆盖图像、视频和医学数据集
- 主流模型在否定查询上召回率仅达随机水平，多选题准确率偏低
- 通过合成否定语料微调CLIP，可提升10%召回率和28%准确率
许多实际视觉语言应用需要模型理解否定，例如用自然语言检索包含某些对象但不包含其他对象的图像。尽管通过大规模训练提升了视觉语言模型（VLMs）性能，其对否定的理解能力仍缺乏深入研究。本研究探讨：当前VLMs对否定的理解程度如何？我们提出NegBench，一个新基准，用于评估18种任务变体下跨图像、视频和医疗数据集的否定理解能力，共包含79,000个样本。该基准包含两个核心任务：带否定的检索与带否定描述的多选题。评估结果表明，现代VLMs在否定理解上表现极差，常处于随机水平。为解决此问题，我们探索数据驱动方法，对CLIP模型在含数百万条否定描述的合成数据集上进行微调。结果表明，该方法可使否定查询的召回率提升10%，多选题准确率提高28%。
原文摘要 · Abstract (English)
Many practical vision-language applications require models that understand negation, e.g., when using natural language to retrieve images which contain certain objects but not others. Despite advancements in vision-language models (VLMs) through large-scale training, their ability to comprehend negation remains underexplored. This study addresses the question: how well do current VLMs understand negation? We introduce NegBench, a new benchmark designed to evaluate negation understanding across 18 task variations and $79$k examples spanning image, video, and medical datasets. The benchmark consists of two core tasks designed to evaluate negation understanding in diverse multimodal settings: Retrieval with Negation and Multiple Choice Questions with Negated Captions. Our evaluation reveals that modern VLMs struggle significantly with negation, often performing at chance level. To address these shortcomings, we explore a data-centric approach wherein we finetune CLIP models on large-scale synthetic datasets containing millions of negated captions. We show that this approach can result in a 10% increase in recall on negated queries and a 28% boost in accuracy on multiple-choice questions with negated captions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。