解决视觉语言预训练中的假负样本问题，提升模型对齐效果。
FALCON: False-Negative Aware Learning of Contrastive Negatives in Vision-Language Alignment
- 动态调整负样本难度，避免假负样本干扰
- 在三种框架上均显著提升下游任务性能
- 适合需要高精度图文对齐的场景
假负样本因大规模数据集中图像与文本的多对多对应关系而成为视觉语言预训练（VLP）的关键挑战。这些假负样本引入冲突的监督信号，破坏学习到的嵌入空间并削弱硬负样本采样的有效性。本文提出FALCON（False-negative Aware Learning of COntrastive Negatives），一种基于学习的迷你批次构建策略，可自适应平衡硬负样本与假负样本之间的权衡。不同于固定启发式方法，FALCON通过一个负样本挖掘调度器，在迷你批次构建过程中动态为每个锚点实例选择适当难度的负样本，其依据是跨模态对齐改进的代理指标。实验表明，FALCON在三种视觉语言学习框架（ALBEF、BLIP-2、SigLIP-2）和广泛的下游任务及评估设置中均显著提升性能，证明了其在缓解假负样本影响方面的有效性和鲁棒性。
原文摘要 · Abstract (English)
False negatives pose a critical challenge in vision-language pretraining (VLP) due to the many-to-many correspondence between images and texts in large-scale datasets. These false negatives introduce conflicting supervision signals that degrade the learned embedding space and diminish the effectiveness of hard negative sampling. In this paper, we propose FALCON (False-negative Aware Learning of COntrastive Negatives), a learning-based mini-batch construction strategy that adaptively balances the trade-off between hard and false negatives during VLP. Rather than relying on fixed heuristics, FALCON employs a negative mining scheduler that dynamically selects negative samples of appropriate hardness for each anchor instance during mini-batch construction, guided by a proxy for cross-modal alignment improvement. Experimental results demonstrate that FALCON significantly improves performance across three vision-language learning frameworks (ALBEF, BLIP-2, SigLIP-2) and a broad range of downstream tasks and evaluation settings, underscoring its effectiveness and robustness in mitigating the impact of false negatives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。