arXiv:2505.16886cs.IRcs.AI2025-05被引 4

推理未必有用,简单模型反而更擅长文档重排序。

Don't "Overthink" Passage Reranking: Is Reasoning Truly Necessary?

  • 对比有无推理步骤的重排序模型,发现无推理更有效。
  • 去掉推理环节后,模型准确率反而提升,说明推理会干扰判断。
  • 理由是推理让评分过于极端,忽略了部分相关性这一关键因素。

随着推理型模型在复杂自然语言任务中表现优异,信息检索领域开始探索将类似推理能力融入基于大语言模型(LLM)的段落重排序器。这类方法通常使用 LLM 生成显式的分步推理过程,再得出最终相关性判断。但推理是否真能提升重排序准确率?本文在相同训练条件下,对比了基于推理的点对点重排序器(ReasonRR)与标准非推理型重排序器(StandardRR),发现 StandardRR 通常表现更优。进一步禁用 ReasonRR 的推理过程(ReasonRR-NoReason)后,结果表明其效果甚至优于原始 ReasonRR。分析显示,推理机制受限于 LLM 的推理过程,导致相关性评分趋于极端化,从而忽略段落的部分相关性,而这正是点对点重排序器准确性的关键因素。

原文摘要 · Abstract (English)

With the growing success of reasoning models across complex natural language tasks, researchers in the Information Retrieval (IR) community have begun exploring how similar reasoning capabilities can be integrated into passage rerankers built on Large Language Models (LLMs). These methods typically employ an LLM to produce an explicit, step-by-step reasoning process before arriving at a final relevance prediction. But, does reasoning actually improve reranking accuracy? In this paper, we dive deeper into this question, studying the impact of the reasoning process by comparing reasoning-based pointwise rerankers (ReasonRR) to standard, non-reasoning pointwise rerankers (StandardRR) under identical training conditions, and observe that StandardRR generally outperforms ReasonRR. Building on this observation, we then study the importance of reasoning to ReasonRR by disabling its reasoning process (ReasonRR-NoReason), and find that ReasonRR-NoReason is surprisingly more effective than ReasonRR. Examining the cause of this result, our findings reveal that reasoning-based rerankers are limited by the LLM's reasoning process, which pushes it toward polarized relevance scores and thus fails to consider the partial relevance of passages, a key factor for the accuracy of pointwise rerankers.

信息检索大模型重排序

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。