让大模型重排文档时考虑上下文，提升复杂查询的检索效果。
Contextual Relevance and Adaptive Sampling for LLM-Based Document Reranking
- 提出'上下文相关性'概念，衡量文档在不同上下文中是否相关
- 实验显示在BRIGHT和BEIR数据集上nDCG@10提升6%-25%
- 设计采样算法TS-SetRank，能自适应选择候选文档组合
重排算法通过高效整合大语言模型（LLM）生成的相关性判断，提升了文档检索质量。然而，对于需要深度推理的查询，识别相关文档仍是重大挑战。这类查询常具有多维度信息需求和细微语义理解，使文档相关性本质上依赖上下文。为此，我们提出‘上下文相关性’，即文档对特定查询的相关概率，对可能出现的多种重排上下文分布（即候选文档集合及其排序）进行边缘化处理。尽管已有研究关注文档顺序带来的位置偏差，我们实证发现，批次组成同样显著影响重排性能。为高效估算上下文相关性，我们提出基于采样的、不确定性感知的重排算法TS-SetRank。实验表明，该方法在BRIGHT和BEIR数据集上相比基线重排模型，nDCG@10提升6%-25%。
原文摘要 · Abstract (English)
Reranking algorithms have made progress in improving document retrieval quality by efficiently aggregating relevance judgments generated by large language models (LLMs). However, identifying relevant documents for queries that require in-depth reasoning remains a major challenge. Reasoning-intensive queries often exhibit multifaceted information needs and nuanced interpretations, rendering document relevance inherently context dependent. To address this, we propose contextual relevance, which we define as the probability that a document is relevant to a given query, marginalized over the distribution of different reranking contexts it may appear in (i.e., the set of candidate documents it is ranked alongside and the order in which the documents are presented to a reranking model). While prior works have studied methods to mitigate the positional bias LLMs exhibit by accounting for the ordering of documents, we empirically find that the compositions of these batches also plays an important role in reranking performance. To efficiently estimate contextual relevance, we propose TS-SetRank, a sampling-based, uncertainty-aware reranking algorithm. Empirically, TS-SetRank improves nDCG@10 over retrieval and reranking baselines by 15-25% on BRIGHT and 6-21% on BEIR, highlighting the importance of modeling relevance as context-dependent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。