用固定参考文档让大模型零样本排序,效率高且准确。
Leveraging Reference Documents for Zero-Shot Ranking via Large Language Models
- 用一个代表查询意图的参考文档,对比所有候选文档
- 在多个数据集上优于独立评分法,接近成对比较法效果
- 适合追求高效准确排序的应用场景
大语言模型在信息检索的文本排序任务中表现优异。点对点方法虽计算高效,但因缺乏文档间比较,易产生偏差;成对方法虽精度高,但需 $O(n^2)$ 复杂度。为此,我们提出 extbf{RefRank},基于固定参考文档进行比较排序:不比较所有文档对,而是将每个候选文档与同一参考锚点对比。通过选择体现核心查询意图的参考文档,隐式捕捉相关性线索,实现文档间的间接比较。该方法将计算复杂度降至线性 $O(n)$,同时保留比较评估优势。为进一步提升鲁棒性,采用加权平均策略融合多参考选择的结果。在多个基准数据集和多种 LLM 上的实验表明,RefRank 显著优于点对点基线,性能至少可媲美成对方法,且计算成本大幅降低。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated exceptional performance in the task of text ranking for information retrieval. While Pointwise ranking approaches offer computational efficiency by scoring documents independently, they often yield biased relevance estimates due to the lack of inter-document comparisons. In contrast, Pairwise methods improve ranking accuracy by explicitly comparing document pairs, but suffer from substantial computational overhead with quadratic complexity ($O(n^2)$). To address this tradeoff, we propose \textbf{RefRank}, a simple and effective comparative ranking method based on a fixed reference document. Instead of comparing all document pairs, RefRank prompts the LLM to evaluate each candidate relative to a shared reference anchor. By selecting the reference anchor that encapsulates the core query intent, RefRank implicitly captures relevance cues, enabling indirect comparison between documents via this common anchor. This reduces computational cost to linear time ($O(n)$) while importantly, preserving the advantages of comparative evaluation. To further enhance robustness, we aggregate multiple RefRank outputs using a weighted averaging scheme across different reference choices. Experiments on several benchmark datasets and with various LLMs show that RefRank significantly outperforms Pointwise baselines and could achieve performance at least on par with Pairwise approaches with a significantly lower computational cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。