用模型错判结果生成反事实样本,提升视觉语言检索精度
RePair: Turning Retrieval Failures into Counterfactual Hard Pairs

- 从错误匹配中提取反事实骨架,通过局部修正生成硬正例
- 仅用10.7万合成样本即超越基线,数据效率提升26%~75%
- 适合追求高效微调的多模态检索研究者
基于CLIP类双编码器的视觉语言检索虽表现优异,但实际精度常依赖于局部语义差异——最相近的误匹配与真匹配仅差一个关键细节。传统难样本挖掘无法构建修正后的对应项;合成增强虽可生成新样本,却缺乏对真实模型失败的条件约束,易扰动无关维度。我们观察到,排名靠前的假阳性可作为反事实模板:其与查询共享大部分语义,仅在导致失败的局部残差上不同。最小化修正该残差,即可得到与真匹配同模态的硬正例;修正版与原始版构成横跨决策边界的硬负对,提供互补的拉-推监督。我们提出RePair,遵循有效性、最小性、局部性三原则,双向挖掘假阳性,使用LLM引导的反事实编辑,并采用局部硬对对比目标训练。在Flickr30K和COCO30K上,RePair仅需10.7万合成样本即优于可控增强基线,比同类方法减少26%~75%数据量,验证了以失败为条件的修复比无差别增强更高效。
原文摘要 · Abstract (English)
Vision-language retrieval with CLIP-style dual encoders achieves strong cross-modal performance, yet practical accuracy often hinges on localized semantic distinctions where top-ranked near misses differ from the true match by a single critical detail. Hard-sample mining can select confusable candidates but cannot construct corrected counterparts; synthetic augmentation can generate novel samples but, without conditioning on actual model failures, targets irrelevant dimensions of hardness. We observe that a top-ranked false positive is a counterfactual scaffold---sharing most of the query's semantics while differing in a localized failure-causing residual. Minimally correcting this residual yields a hard positive of the ground truth in the same modality; the corrected and unedited versions form a hard negative pair that straddles the decision boundary, producing complementary pull--push supervision. We introduce RePair, guided by three principles---Validity, Minimality, and Locality---which mines false positives bidirectionally, applies LLM-guided counterfactual editing, and trains with a local hard-pair contrastive objective. On Flickr30K and COCO30K, RePair outperforms controlled augmentation baselines with only 107K synthetic samples---26\%--75\% fewer than comparable methods---confirming failure-conditioned repair is more data-efficient than error-agnostic augmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。