用可控替换检测描述词对齐程度，筛选更高质量的图文数据。
What Does the Caption Really Say? Counterfactual Phrase Intervention for Compositional Data Selection in Vision-Language Pretraining
- 通过替换描述中的词并观察图像-文本得分变化，评估每个词的重要性。
- 在CC3M规模上，用该方法选50%数据，关系识别性能提升1.91点。
- 不依赖具体模型损失，可通用提升多种视觉语言模型表现。
CLIP类对比预训练通常使用样本级过滤信号（如配对对齐度）来筛选网络规模的图文对。我们发现该信号存在饱和现象：一旦粗粒度错配被移除，更严格的全局过滤无法反映保留描述所提供的组合性监督。根本原因在于全局分数混淆了图文整体合理性与描述中物体、属性、关系短语是否真正支撑匹配。后者才是组合泛化所需，但现有配对过滤对此无感。为此提出反事实短语干预（CPI），一种短语级筛选框架：通过受控的随机词替换生成基于图像的短语敏感度评分。CPI仅用全局对齐做粗粒度错配剔除，再根据短语在控制替换下对图文得分的影响程度排序剩余数据。将CPI视为一阶短语敏感信号而非实体定位结果，在CC3M规模验证。按此信号排序的50%数据子集，在相同预算下使VL-CheckList-VG关系任务提升+1.91（相比全量基线）和+1.00（相比仅对齐过滤），同时提升SugarCrepe整体表现并保持迁移能力。CPI与损失函数正交：在NegCLIP上直接应用仍带来+3.84的额外增益，主文中亦见CE-CLIP提升。
原文摘要 · Abstract (English)
CLIP-style contrastive pretraining typically curates web-scale image-text pairs using sample-level filtering signals, often based on pair-level alignment. We show that this signal saturates: once coarse mismatches are removed, stricter global filtering no longer tracks the compositional supervision provided by the retained captions. The reason is structural - a global score conflates whether a pair is broadly plausible with whether the individual object, attribute, and relation phrases inside the caption materially support the image-text match. The latter is what compositional generalization demands, yet pair-level filters are blind to it. We address this with Counterfactual Phrase Intervention (CPI), a phrase-level curation framework that converts controlled nonce-token substitutions into image-conditioned phrase-sensitivity scores. CPI uses global alignment only for coarse mismatch removal, then ranks the surviving pool by whether caption phrases measurably affect the image-text score under controlled substitution. We frame CPI as a first-order phrase-sensitivity signal rather than a grounding or identification result, and evaluate it at CC3M scale. Ranking by this signal yields a 50%-data subset that improves VL-CheckList-VG Relation by +1.91 over the full-data baseline and +1.00 over alignment-only filtering at matched budget, while improving SugarCrepe overall and preserving general transfer. CPI is loss-orthogonal: applied unchanged to NegCLIP, it further improves VL-CheckList-VG Relation by +3.84, with additional CE-CLIP gains in the main text.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。