用场景图引导的CLIP模型提升组合推理鲁棒性
CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning
- 通过场景图识别组合元素，构造结构化负样本
- 在多种组合元素上表现优于现有方法，且泛化能力强
- 训练样本少，适合需要高效学习的视觉语言任务
视觉语言模型在组合推理基准测试中表现优异，但我们的控制分析发现，现有组合感知模型存在特定元素偏差，某些情况下甚至不如原始CLIP。为此，我们提出基于场景图引导的组合式CLIP（CS-CLIP），利用场景图识别组合元素，并通过选择性掩码构建结构化负样本。同时保留与原描述最矛盾的负样本，迫使模型依赖组合结构而非表面线索。CS-CLIP在组合推理任务中达到当前最优性能，跨组合元素表现稳健，且保持了跨模态检索和下游视觉推理能力，训练样本需求低于先前方法。
原文摘要 · Abstract (English)
Vision-language models (VLMs) demonstrate strong performance across compositional reasoning benchmarks, which require reasoning over semantic perturbations of objects, attributes, relations, and their interactions. However, our controlled analysis reveals that existing compositionality-aware VLMs exhibit element-specific biases, often underperforming vanilla CLIP on certain compositional elements. To address this, we propose Compositional Scene Graph-guided CLIP (CS-CLIP), which uses scene graphs to identify compositional elements and construct structured negatives via selective masking. We further retain negatives that are most contradictory to the original caption, forcing the model to rely on compositional structure rather than surface cues. CS-CLIP achieves state-of-the-art compositional reasoning with robust performance across compositional elements. It also preserves general vision-language capabilities such as cross-modal retrieval and downstream visual reasoning, while requiring fewer training samples than prior methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。