arXiv:2506.21638cs.IRcs.AI2025-06

用强化学习让大模型更会推理排序,性能超越更大模型。

R1-Ranker: Teaching LLM Rankers to Reason

  • 设计双架构:单步生成与分步迭代,用奖励鼓励深度推理。
  • 30亿参数模型在9个数据集上平均提升15.7%,超70亿模型。
  • 适合需要复杂推理的推荐、检索等场景,可复用推理过程。

大型语言模型(LLMs)在数学、编程和科学问题解决等领域展现出强大的推理能力,但在排序任务(如信息检索、推荐系统和大模型路由)中的潜力仍待挖掘。排序需对异构候选者进行复杂推理,但现有基于LLM的排序器往往领域特定、依赖固定骨干网络,且缺乏迭代优化,难以充分发挥LLM的推理潜能。为此,我们提出R1-Ranker,一种基于强化学习的推理激励框架,包含两种互补设计:DRanker一次性生成完整排名,IRanker将排序分解为逐步淘汰过程,并通过每一步的奖励促进深层推理。我们在涵盖推荐、路由和段落排序的九个数据集上评估统一的R1-Rankers,结果显示IRanker-3B持续达到领先性能,在部分任务上超越更大的7B模型,平均相对提升达15.7%。消融与泛化实验进一步证实强化学习和迭代推理的关键作用,其中IRanker-3B在跨域任务上的零样本性能提升超过9%,其推理轨迹还能使其他LLM性能最高提升22.87%。结果表明,以单一推理驱动基础模型统一多样排序任务,不仅有效,更是推动大模型在排序场景中推理能力发展的关键。

原文摘要 · Abstract (English)

Large language models (LLMs) have recently shown strong reasoning abilities in domains like mathematics, coding, and scientific problem-solving, yet their potential for ranking tasks, where prime examples include retrieval, recommender systems, and LLM routing, remains underexplored. Ranking requires complex reasoning across heterogeneous candidates, but existing LLM-based rankers are often domain-specific, tied to fixed backbones, and lack iterative refinement, limiting their ability to fully exploit LLMs' reasoning potential. To address these challenges, we propose R1-Ranker, a reasoning-incentive framework built on reinforcement learning, with two complementary designs: DRanker, which generates full rankings in one shot, and IRanker, which decomposes ranking into an iterative elimination process with step-wise rewards to encourage deeper reasoning. We evaluate unified R1-Rankers on nine datasets spanning recommendation, routing, and passage ranking, showing that IRanker-3B consistently achieves state-of-the-art performance, surpasses larger 7B models on some tasks, and yields a 15.7% average relative improvement. Ablation and generalization experiments further confirm the critical role of reinforcement learning and iterative reasoning, with IRanker-3B improving zero-shot performance by over 9% on out-of-domain tasks and reasoning traces boosting other LLMs by up to 22.87%. These results demonstrate that unifying diverse ranking tasks with a single reasoning-driven foundation model is both effective and essential for advancing LLM reasoning in ranking scenarios.

推理排序强化学习大模型推荐系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。