用生成式推理重排提升推荐系统精准度,解决传统方法忽视重排与可扩展性难题。
GR2: Generative Reasoning Re-ranker
- 通过语义ID编码和三阶段训练,让大模型在重排阶段发挥深度推理能力。
- 在真实数据集上比SOTA高出2.4%召回率与1.3%NDCG,验证有效性。
- 设计可验证奖励机制防止模型作弊,适合工业级推荐系统优化场景。
近期研究将大语言模型(LLM)引入推荐系统,因其可扩展性和世界知识优势。但现有工作存在三大局限:(1)多聚焦于检索与排序,忽视关键的重排阶段;(2)通常采用零样本或监督微调,未充分利用强化学习与高质量推理数据带来的推理能力;(3)物品常以非语义ID表示,在包含数十亿项的工业系统中面临严重可扩展性挑战。为此,我们提出生成式推理重排器(GR2),一个面向重排的端到端框架,包含三阶段训练流程:首先,使用编码器将非语义ID转换为语义ID(唯一性≥99%)并进行中段训练;其次,用更大规模的LLM通过精心设计提示与拒绝采样生成高质量推理轨迹,用于监督微调以赋予基础推理能力;最后,采用解耦的Clip与动态采样策略优化(DAPO),实现可扩展的强化学习监督,并设计专为重排定制的可验证奖励。在两个真实数据集上的实验表明,GR2在Recall@5上优于SOTA OneRec-Think 2.4%,在NDCG@5上提升1.3%。消融实验确认高级推理轨迹带来显著性能提升。进一步发现,奖励设计对重排至关重要:若无约束,模型易通过保持物品顺序进行奖励欺骗,因此采用条件可验证奖励可有效抑制该行为并优化性能。
原文摘要 · Abstract (English)
Recent studies increasingly explore Large Language Models (LLMs) as a new paradigm for recommendation systems due to their scalability and world knowledge. However, existing work has three key limitations: (1) most efforts focus on retrieval and ranking, while the reranking phase, critical for refining final recommendations, is largely overlooked; (2) LLMs are typically used in zero-shot or supervised fine-tuning settings, leaving their reasoning abilities, especially those enhanced through reinforcement learning (RL) and high-quality reasoning data, underexploited; (3) items are commonly represented by non-semantic IDs, creating major scalability challenges in industrial systems with billions of identifiers. To address these gaps, we propose the Generative Reasoning Reranker (GR2), an end-to-end framework with a three-stage training pipeline tailored for reranking. First, a pretrained LLM is mid-trained on semantic IDs encoded from non-semantic IDs via a tokenizer achieving $\ge$99% uniqueness. Next, a stronger larger-scale LLM generates high-quality reasoning traces through carefully designed prompting and rejection sampling, which are used for supervised fine-tuning to impart foundational reasoning skills. Finally, we apply Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO), enabling scalable RL supervision with verifiable rewards designed specifically for reranking. Experiments on two real-world datasets demonstrate GR2's effectiveness: it surpasses the state-of-the-art OneRec-Think by 2.4% in Recall@5 and 1.3% in NDCG@5. Ablations confirm that advanced reasoning traces yield substantial gains across metrics. We further find that RL reward design is crucial in reranking: LLMs tend to exploit reward hacking by preserving item order, motivating conditional verifiable rewards to mitigate this behavior and optimize reranking performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。