arXiv:2411.00142cs.CLcs.AI2024-11被引 39

用大模型模拟人类思考,精准重排长文档。

JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking

  • 分三步分析问题、提取摘要、判断相关性,模仿人类推理。
  • 在BRIGHT基准上显著优于基础检索,媲美微调顶尖模型。
  • 适合作为通用零样本重排器,支持不同规模模型集成。

准确的文档检索对检索增强生成(RAG)应用至关重要,包括开放域问答和代码补全。尽管大语言模型(LLMs)已被用于RAG系统中的密集编码或列表式重排,但在需要复杂推理的任务中表现不佳,因缺乏对文档相关性的细致分析。为此,我们提出JudgeRank,一种新型代理式重排器,模拟人类评估文档相关性的认知过程。该方法包含三个关键步骤:(1)查询分析以识别核心问题,(2)文档分析以提取与查询相关的摘要,(3)相关性判断以提供简洁的评估。我们在推理密集型的BRIGHT基准上评估JudgeRank,结果表明其性能显著优于第一阶段检索方法,并超越其他主流重排方法。此外,JudgeRank在流行的BEIR基准上表现与微调的顶尖重排器相当,验证了其零样本泛化能力。通过全面的消融实验,我们证明了JudgeRank在不同规模的LLMs上均具备良好泛化性,且模型集成能带来更精确的重排效果。

原文摘要 · Abstract (English)

Accurate document retrieval is crucial for the success of retrieval-augmented generation (RAG) applications, including open-domain question answering and code completion. While large language models (LLMs) have been employed as dense encoders or listwise rerankers in RAG systems, they often struggle with reasoning-intensive tasks because they lack nuanced analysis when judging document relevance. To address this limitation, we introduce JudgeRank, a novel agentic reranker that emulates human cognitive processes when assessing document relevance. Our approach consists of three key steps: (1) query analysis to identify the core problem, (2) document analysis to extract a query-aware summary, and (3) relevance judgment to provide a concise assessment of document relevance. We evaluate JudgeRank on the reasoning-intensive BRIGHT benchmark, demonstrating substantial performance improvements over first-stage retrieval methods and outperforming other popular reranking approaches. In addition, JudgeRank performs on par with fine-tuned state-of-the-art rerankers on the popular BEIR benchmark, validating its zero-shot generalization capability. Through comprehensive ablation studies, we demonstrate that JudgeRank's performance generalizes well across LLMs of various sizes while ensembling them yields even more accurate reranking than individual models.

重排大模型推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。