arXiv:2411.04602cs.IRcs.CL2024-11被引 18

用大模型全局重排文档,提升排序效率和准确性

Self-Calibrated Listwise Reranking with Large Language Models

  • 引入显式列表级相关性评分,实现全集全局比较
  • 通过模型自生成点级评分校准列表级分数,保证可比性
  • 在BEIR和TREC数据集上验证了高效且准确

大型语言模型(LLMs)凭借强大的语言能力,已通过序列到序列方法应用于重排任务。该方法以列表方式重排多个段落,并生成文本形式的重排结果。然而,由于LLM上下文窗口有限,需采用滑动窗口策略迭代处理大规模候选集,不仅增加计算成本,也限制模型充分捕捉所有候选项间的对比信息。为此,我们提出一种新颖的自校准列表级重排方法,旨在利用LLM生成全局相关性得分。首先,设计相关性感知的列表级重排框架,引入显式列表视图相关性评分,提升重排效率并支持全集全局比较。其次,为确保评分可比性,提出自校准训练,使用模型内部生成的点级相关性评估来校准列表级评分。在BEIR基准和TREC深度学习赛道上的大量实验与全面分析证明了该方法的有效性与高效性。

原文摘要 · Abstract (English)

Large language models (LLMs), with advanced linguistic capabilities, have been employed in reranking tasks through a sequence-to-sequence approach. In this paradigm, multiple passages are reranked in a listwise manner and a textual reranked permutation is generated. However, due to the limited context window of LLMs, this reranking paradigm requires a sliding window strategy to iteratively handle larger candidate sets. This not only increases computational costs but also restricts the LLM from fully capturing all the comparison information for all candidates. To address these challenges, we propose a novel self-calibrated listwise reranking method, which aims to leverage LLMs to produce global relevance scores for ranking. To achieve it, we first propose the relevance-aware listwise reranking framework, which incorporates explicit list-view relevance scores to improve reranking efficiency and enable global comparison across the entire candidate set. Second, to ensure the comparability of the computed scores, we propose self-calibrated training that uses point-view relevance assessments generated internally by the LLM itself to calibrate the list-view relevance assessments. Extensive experiments and comprehensive analysis on the BEIR benchmark and TREC Deep Learning Tracks demonstrate the effectiveness and efficiency of our proposed method.

大模型重排相关性评分自校准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。