用大模型识别并修正训练数据中的错标相关项，显著提升检索模型性能。
Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMs
- 用大模型自动发现并重标训练数据中的假负例（应相关却被标为无关）
- 使E5和Qwen2.5模型在BEIR上提升0.7~1.4点，零样本测试提升1.7~1.8点
- 方法简单高效，适合关注检索质量与数据清洗的研究者
训练鲁棒的检索与重排序模型通常依赖大规模检索数据集，例如BGE包含160万条查询-段落对。然而我们发现，某些数据集反而会损害模型效果——从BGE中移除15个数据集中的8个，训练集规模缩小2.35倍，却使BEIR上的nDCG@10提升1.0点。这促使我们深入考察训练数据质量，重点关注“假负例”问题：本应相关却被错误标注为无关的段落。我们采用大模型作为低成本、高效的手段，识别并重标这些假负例。实验表明，将假负例重标为真正例后，E5（base）和Qwen2.5-7B检索模型在BEIR上分别提升0.7~1.4点，在零样本AIR-Bench上提升1.7~1.8点；在重标数据上微调的重排序器（如Qwen2.5-3B）也获得类似提升。人工标注验证了大模型判断的可靠性。训练数据与代码已公开。
原文摘要 · Abstract (English)
Training robust retrieval and reranker models typically relies on large-scale retrieval datasets; for example, the BGE collection contains 1.6 million query-passage pairs sourced from various data sources. However, we find that certain datasets can negatively impact model effectiveness -- pruning 8 out of 15 datasets from the BGE collection, reduces the training set size by 2.35$\times$, surprisingly increases nDCG@10 on BEIR by 1.0 point. This motivates a deeper examination of training data quality, with a particular focus on "false negatives", where relevant passages are incorrectly labeled as irrelevant. We utilize LLMs as a simple, cost-effective approach to identify and relabel false negatives in training datasets. Experimental results show that relabeling false negatives as true positives improves both E5 (base) and Qwen2.5-7B retrieval models by 0.7$\unicode{x2013}$1.4 points on BEIR and by 1.7$\unicode{x2013}$1.8 points at nDCG@10 on zero-shot AIR-Bench evaluation. Similar gains are observed for rerankers fine-tuned on the relabeled data, such as Qwen2.5-3B on BEIR. The reliability of LLMs to identify false negatives is supported by human annotation results. Our training dataset and code are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。