让CLIP无需微调就能理解否定描述，避免误匹配。
Not Just What's There: Enabling CLIP to Comprehend Negated Visual Descriptions Without Fine-tuning
- 用双阶段模块分离否定语义并动态调整相似度计算
- 在跨域测试中优于现有方法，低资源下表现更稳健
- 无需微调，适合快速部署到新场景
视觉-语言模型（如CLIP）难以理解否定句，常将“无狗”与狗图像错误匹配。现有方法通过微调文本编码器改善，但易过拟合。本文提出CLIPGlasses，一种即插即用框架：首先通过镜头模块分离否定语义，再由框架模块预测上下文相关的排斥强度，并融入修改后的相似度计算，抑制与否定语义的对齐，降低误匹配。实验表明，搭载CLIPGlasses的CLIP在域内性能相当，跨域泛化能力超越当前最优方法，尤其在低资源条件下优势显著，体现更强的鲁棒性。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) like CLIP struggle to understand negation, often embedding affirmatives and negatives similarly (e.g., matching "no dog" with dog images). Existing methods refine negation understanding via fine-tuning CLIP's text encoder, risking overfitting. In this work, we propose CLIPGlasses, a plug-and-play framework that enhances CLIP's ability to comprehend negated visual descriptions. CLIPGlasses adopts a dual-stage design: a Lens module disentangles negated semantics from text embeddings, and a Frame module predicts context-aware repulsion strength, which is integrated into a modified similarity computation to penalize alignment with negated semantics, thereby reducing false positive matches. Experiments show that CLIP equipped with CLIPGlasses achieves competitive in-domain performance and outperforms state-of-the-art methods in cross-domain generalization. Its superiority is especially evident under low-resource conditions, indicating stronger robustness across domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。