用强化学习让大模型更会推理排序,性能超越更大模型。
R1-Ranker: Teaching LLM Rankers to Reason
- 设计双架构:单步生成与分步迭代,用奖励鼓励深度推理。
- 30亿参数模型在9个数据集上平均提升15.7%,超70亿模型。
- 适合需要复杂推理的推荐、检索等场景,可复用推理过程。
大型语言模型(LLMs)在数学、编程和科学问题解决等领域展现出强大的推理能力,但在排序任务(如信息检索、推荐系统和大模型路由)中的潜力仍待挖掘。排序需对异构候选者进行复杂推理,但现有基于LLM的排序器往往领域特定、依赖固定骨干网络,且缺乏迭代优化,难以充分发挥LLM的推理潜能。为此,我们提出R1-Ranker,一种基于强化学习的推理激励框架,包含两种互补设计:DRanker一次性生成完整排名,IRanker将排序分解为逐步淘汰过程,并通过每一步的奖励促进深层推理。我们在涵盖推荐、路由和段落排序的九个数据集上评估统一的R1-Rankers,结果显示IRanker-3B持续达到领先性能,在部分任务上超越更大的7B模型,平均相对提升达15.7%。消融与泛化实验进一步证实强化学习和迭代推理的关键作用,其中IRanker-3B在跨域任务上的零样本性能提升超过9%,其推理轨迹还能使其他LLM性能最高提升22.87%。结果表明,以单一推理驱动基础模型统一多样排序任务,不仅有效,更是推动大模型在排序场景中推理能力发展的关键。
原文摘要 · Abstract (English)
Large language models (LLMs) have recently shown strong reasoning abilities in domains like mathematics, coding, and scientific problem-solving, yet their potential for ranking tasks, where prime examples include retrieval, recommender systems, and LLM routing, remains underexplored. Ranking requires complex reasoning across heterogeneous candidates, but existing LLM-based rankers are often domain-specific, tied to fixed backbones, and lack iterative refinement, limiting their ability to fully exploit LLMs' reasoning potential. To address these challenges, we propose R1-Ranker, a reasoning-incentive framework built on reinforcement learning, with two complementary designs: DRanker, which generates full rankings in one shot, and IRanker, which decomposes ranking into an iterative elimination process with step-wise rewards to encourage deeper reasoning. We evaluate unified R1-Rankers on nine datasets spanning recommendation, routing, and passage ranking, showing that IRanker-3B consistently achieves state-of-the-art performance, surpasses larger 7B models on some tasks, and yields a 15.7% average relative improvement. Ablation and generalization experiments further confirm the critical role of reinforcement learning and iterative reasoning, with IRanker-3B improving zero-shot performance by over 9% on out-of-domain tasks and reasoning traces boosting other LLMs by up to 22.87%. These results demonstrate that unifying diverse ranking tasks with a single reasoning-driven foundation model is both effective and essential for advancing LLM reasoning in ranking scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。