arXiv:2606.02814cs.IRcs.AI2026-06

神经检索器会偏好特定文档类型,导致真正相关但非主流内容更难被找到。

Do Neural Retrievers Prefer Certain Documents? Evidence of Learned Relevance Priors

论文配图:Do Neural Retrievers Prefer Certain Documents? Evidence of Learned Relevance Priors
图 1 · 摘自论文原文
  • 通过分析冻结嵌入训练分类器,发现检索器隐含学习了文档层面的偏好信号。
  • 这种偏好导致非主流文档即使相关也更难被检索,形成可量化的可发现性差距。
  • 适合关注检索公平性、数据偏差与模型可解释性的研究者阅读。

神经检索器通过标注的查询-文档对学习相关性判断,但标注协议可能并非纯粹反映相关性:仅选择部分文档进行标注,且这一选择倾向某些文档类型。我们研究监督式双编码器检索器是否隐式学习到文档级相关性先验——即在表示空间中编码的、与查询无关的信号,作为训练标注数据的副作用。通过在冻结文档嵌入上训练简单分类器,并在多个信息检索基准上评估三种先进检索器,我们发现监督式神经检索器确实编码了可泛化至未见文档的相关性先验,且跨模型一致。这些先验造成可发现性差距:低先验文档系统性更难被检索,即便它们实际上相关。该现象出现在监督密集检索器中,但在BM25中较弱且不一致,且在受控匹配文档对比中依然存在。利用大语言模型生成解释,我们发现被标记为相关的文档往往是对主流话题的全面、自包含摘要,而小众、碎片化或高度技术性内容常未被标注。检索器内化此偏见,将具有这些偏好特征的文档排在更高位置,与实际相关性无关。我们的发现揭示了监督检索的结构性局限:模型不仅学习相关性,还学习训练数据中的隐含文档偏好。

原文摘要 · Abstract (English)

Neural retrievers are trained to estimate query-document relevance from annotated query-document pairs. Yet annotation protocols may not purely reflect relevance: they select only a subset of documents for labeling, and this selection can favor certain document types over others. We investigate whether supervised bi-encoder retrievers implicitly learn a document-level relevance prior: a query-independent signal encoded in their representation space as a side effect of training on annotated data. We estimate this prior by training simple classifiers on frozen document embeddings and evaluate three state-of-the-art retrievers across multiple IR benchmarks. We find that supervised neural retrievers encode relevance priors that generalize to unseen documents and are consistent across models. These priors create a findability gap: documents with lower prior are systematically harder to retrieve, even when genuinely relevant. This effect appears in supervised dense retrievers but is weaker and less consistent in BM25, and it persists under controlled matched-document comparisons. Using LLM-based explanations, we find that judged-relevant documents tend to be comprehensive, self-contained summaries of mainstream topics, while niche, fragmentary, or highly technical content is often left unjudged. Retrievers internalize this bias, ranking documents with these favored features higher than documents that lack them, independently of their actual relevance. Our findings expose a structural limitation of supervised retrieval: models trained on annotated data do not just learn relevance, but also the implicit document preferences in their training data.

信息检索模型偏差检索公平性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。