修正InfoNCE的极端值偏差，提升对比学习性能
When Softmax Fails at the Top: Extreme Value Corrections for InfoNCE
- 用极值理论分析软最大值假设的偏差，提出改进方法
- 在5个视觉基准上，冻结特征评估均实现一致提升
- 无需额外参数，适合希望即插即用提升效果的研究者
InfoNCE是标准的对比学习目标，但其软最大值形式不仅出于计算便利，还隐含了对最优样本选择方式的统计假设。通过极值理论分析发现，该假设常与现代对比学习中归一化嵌入设置不匹配。为此，我们提出 extsc{WEINCE}，一种简单修改：利用锚点相关的在线批次统计量，将普通软最大值logits与端点不足修正项融合，不引入可训练参数。在五个视觉基准上， extsc{WEINCE}在冻结特征评估中均取得一致性能提升。结果表明，对困难负样本更精确的统计处理能有效改善对比学习目标。
原文摘要 · Abstract (English)
InfoNCE is the standard contrastive learning objective, but its softmax form is not only a computational convenience: it also encodes a statistical assumption about how the top-scoring example is selected. Using extreme value theory, we show that this assumption is often misaligned with the normalized embedding setting used in modern contrastive learning. Motivated by this mismatch, we propose \textsc{WEINCE}, a simple modification of InfoNCE that uses anchor-wise online batch statistics to blend the usual softmax logits with an endpoint shortfall correction, adding no trainable parameters. Across five vision benchmarks, \textsc{WEINCE} yields consistent improvements in frozen-feature evaluation. These results show that a more faithful statistical treatment of hard negatives can improve contrastive objectives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。