arXiv:2510.08985cs.IR2025-10被引 10

大模型推理对文档重排序无效,反而增加成本且降低精度。

Rethinking Reasoning in Document Ranking: Why Chain-of-Thought Falls Short

  • 对比直接预测与带思维链的重排序方法
  • 带推理的模型在多个数据集上表现更差,误报率上升
  • 适合关注模型稳定性和效率的研究者

文档重排序是信息检索中的关键环节,旨在优化初始结果以提升下游任务质量。近期研究受大模型启发,尝试在基于大语言模型的重排序器中引入显式思维链(CoT)推理。然而,这种推理在重排序任务中的有效性尚未充分探索。本文首次系统性地研究了点对点与列表式重排序设置下,监督微调与强化学习场景中的推理效果。基于BRIGHT等推理密集型数据集和BEIR等标准信息检索基准,发现加入推理的重排序器始终劣于不使用推理的直接预测模型,且推理成本显著更高。分析揭示三大问题:(i) 点对点模型中,推理破坏校准性,使模型偏向正类,提高真阳性率但降低真阴性率,导致负样本主导池中误报激增;(ii) 列表式模型中,推理虽提升域内拟合度,但增加方差,无法外推到新领域,即使强化学习缩短推理链也无改善;(iii) 直接微调的重排序器始终更稳定、有效、鲁棒。该研究挑战了‘显式推理普遍有益’的假设,提出未来方向包括点对点模型的校准感知评分,以及设计简洁、精准的推理策略以缓解过拟合与过度思考问题。

原文摘要 · Abstract (English)

Document reranking is a key component in information retrieval (IR), aimed at refining initial retrieval results to improve ranking quality for downstream tasks. Recent studies--motivated by large reasoning models (LRMs)--have begun incorporating explicit chain-of-thought (CoT) reasoning into LLM-based rerankers. However, the effectiveness of such reasoning for ranking tasks remains underexplored. In this work, we present the first systematic study of reasoning in reranking across both pointwise and listwise settings, under both supervised fine-tuning and reinforcement learning. Using diverse benchmarks, including reasoning-intensive datasets (BRIGHT) and standard IR benchmarks (BEIR), we find that reasoning-augmented rerankers consistently underperform their direct counterparts that predict rankings without CoT, despite substantially higher inference costs. Our analysis reveals three core limitations: (i) in pointwise rerankers, reasoning breaks calibration and biases models toward the positive class, raising TPR but lowering TNR, which inflates false positives and degrades ranking in negative-dominant pools; (ii) in listwise rerankers, reasoning improves in-domain fit but increases variance and fails to generalize out-of-domain, even when reinforcement learning shortens rationales; and (iii) overall, directly fine-tuned rerankers remain more stable, effective, and robust. These findings challenge the assumption that explicit reasoning is universally beneficial for reranking. We conclude by highlighting future directions, including calibration-aware scoring for pointwise rerankers and the design of concise, targeted reasoning strategies to mitigate overfitting and overthinking in listwise rerankers.

信息检索大模型重排序推理缺陷

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。