用AUC优化检索模型,让相关文档得分更高更可靠。
Optimizing What Matters: AUC-Driven Learning for Robust Neural Retrieval
- 新损失函数直接优化AUC,提升正负样本排序能力
- 在AUC和标准指标上均显著优于对比学习方法
- 适合对排序可靠性要求高的生成式检索场景
双编码器检索器依赖相关文档得分高于无关文档的原则。然而主流的噪声对比估计(NCE)目标,即对比损失的基础,优化的是一个软化排序代理,我们严格证明其本质上对得分分离质量无感,且与AUC无关。这一错配导致下游任务如检索增强生成(RAG)中校准性差、性能不佳。为此,我们提出MW损失,该损失最大化曼-惠特尼U统计量,数学上等价于受试者工作特征曲线下面积(AUC)。MW损失通过最小化得分差的二元交叉熵,促使每对正负样本正确排序。我们提供理论保证:MW损失直接上界于AoC,更贴合检索目标。进一步提倡使用ROC曲线和AUC作为无需设定阈值的评估工具,以诊断检索器校准性和排序质量。实验证明,使用MW损失训练的检索器在AUC及标准检索指标上持续优于对比学习方法,生成更可靠、更具区分性的检索结果,适用于高风险场景如RAG。
原文摘要 · Abstract (English)
Dual-encoder retrievers depend on the principle that relevant documents should score higher than irrelevant ones for a given query. Yet the dominant Noise Contrastive Estimation (NCE) objective, which underpins Contrastive Loss, optimizes a softened ranking surrogate that we rigorously prove is fundamentally oblivious to score separation quality and unrelated to AUC. This mismatch leads to poor calibration and suboptimal performance in downstream tasks like retrieval-augmented generation (RAG). To address this fundamental limitation, we introduce the MW loss, a new training objective that maximizes the Mann-Whitney U statistic, which is mathematically equivalent to the Area under the ROC Curve (AUC). MW loss encourages each positive-negative pair to be correctly ranked by minimizing binary cross entropy over score differences. We provide theoretical guarantees that MW loss directly upper-bounds the AoC, better aligning optimization with retrieval goals. We further promote ROC curves and AUC as natural threshold free diagnostics for evaluating retriever calibration and ranking quality. Empirically, retrievers trained with MW loss consistently outperform contrastive counterparts in AUC and standard retrieval metrics. Our experiments show that MW loss is an empirically superior alternative to Contrastive Loss, yielding better-calibrated and more discriminative retrievers for high-stakes applications like RAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。