发现密集检索模型受短文档、早期位置等偏见影响，严重忽略事实证据。
Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence
- 通过控制实验揭示检索模型偏好短文档、早期位置和字面匹配。
- 多偏见叠加时，正确答案文档召回率低于10%。
- 该偏见导致RAG下游性能下降34%，比不提供文档更差。
密集检索模型广泛用于信息检索任务，如检索增强生成（RAG）。由于其在系统中常处于首步，鲁棒性至关重要。本文利用关系抽取数据集（如Re-DocRED）设计可控实验，量化了短文档偏好等启发式偏见对Dragon+和Contriever等检索模型的影响。研究发现，模型显著偏好短文档、早期位置、重复实体和字面匹配，却忽视答案存在。当多种偏见叠加时，模型在合成无答案偏见文档上选择含答案文档的比例不足10%。此外，这些偏见直接影响下游应用，如RAG中，被优先检索的文档会误导大语言模型，导致性能比不提供文档时还低34%。
原文摘要 · Abstract (English)
Dense retrieval models are commonly used in Information Retrieval (IR) applications, such as Retrieval-Augmented Generation (RAG). Since they often serve as the first step in these systems, their robustness is critical to avoid downstream failures. In this work, we repurpose a relation extraction dataset (e.g., Re-DocRED) to design controlled experiments that quantify the impact of heuristic biases, such as a preference for shorter documents, on retrievers like Dragon+ and Contriever. We uncover major vulnerabilities, showing retrievers favor shorter documents, early positions, repeated entities, and literal matches, all while ignoring the answer's presence! Notably, when multiple biases combine, models exhibit catastrophic performance degradation, selecting the answer-containing document in less than 10% of cases over a synthetic biased document without the answer. Furthermore, we show that these biases have direct consequences for downstream applications like RAG, where retrieval-preferred documents can mislead LLMs, resulting in a 34% performance drop than providing no documents at all. https://huggingface.co/datasets/mohsenfayyaz/ColDeR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。