不微调模型就能让视觉语言模型更好理解否定句
SpaceVLM: Sub-Space Modeling of Negation in Vision-Language Models
- 把否定看作嵌入空间中的一个子区域，而非单一位置
- 在检索、多项选择和图文生成任务上平均提升30%的否定理解能力
- 无需微调，保持原始模型零样本性能，适合实际部署
视觉语言模型在处理否定句时表现不佳。例如面对‘检索（或生成）没有行人的街道场景’这类提示时，常忽略‘无’字。现有方法通过在大规模否定数据集上微调来改进，但会损害模型在肯定句上的零样本性能。我们发现如CLIP等模型的嵌入空间可划分为语义一致的子空间。基于此，提出无需训练的框架，将否定建模为联合嵌入空间中的一个子空间，而非单一点。对于‘A但不是N’类描述，构建A与N对应嵌入的球冠区域，通过靠近A且远离N的中心方向评分图像。在检索、多项选择和文本到图像任务中，该方法平均比之前方法提升约30%的否定理解能力，弥合了肯定与否定提示间的差距，同时保留了未微调模型的零样本性能。代码将在发表后公开。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) struggle with negation. Given a prompt like "retrieve (or generate) a street scene without pedestrians," they often fail to respect the "not." Existing methods address this limitation by fine-tuning on large negation datasets, but such retraining often compromises the model's zero-shot performance on affirmative prompts. We show that the embedding space of VLMs, such as CLIP, can be divided into semantically consistent subspaces. Based on this property, we propose a training-free framework that models negation as a subspace in the joint embedding space rather than a single point (Figure 1). To find the matching image for a caption such as "A but not N," we construct two spherical caps around the embeddings of A and N, and we score images by the central direction of the region that is close to A and far from N. Across retrieval, MCQ, and text-to-image tasks, our method improves negation understanding by about 30% on average over prior methods. It closes the gap between affirmative and negated prompts while preserving the zero-shot performance that fine-tuned models fail to maintain. Code will be released upon publication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。