用大模型提升推荐系统最终排序效果,显著改善用户点击率。
GR2 Technical Report

- 用语义ID和推理轨迹蒸馏,让大模型理解推荐上下文。
- 强化学习优化排序,工业级流量下点击率提升超18%。
- 适合大规模推荐系统,尤其关注最终排序体验的团队。
工业级推荐系统通过多阶段流程服务数十亿用户,其中最终重排序阶段对用户参与度和下游性能影响最大,尤其在轮播与网格展示中。尽管大语言模型在推荐领域备受关注,但存在三大障碍:(1)多数研究集中于召回与排序,忽视最贴近用户交互的重排序阶段;(2)大模型通常零样本或监督微调,未充分释放强化学习在可验证奖励下的推理能力;(3)部署商品库包含数十亿非语义标识符,超出基础大模型词汇范围。本文提出GR2(生成式推理重排序器),端到端框架结合:(i)使用≥99%唯一性的语义ID进行中期训练;(ii)通过定向提示与拒绝采样从更强教师模型中蒸馏推理轨迹;(iii)为重排序定制的可验证奖励强化学习。为实现资源可行,进一步引入上下文压缩器,采用可扩展的在线蒸馏替代监督微调(发现其在工业规模下失效),以及用于低延迟推理的推理蒸馏。GR2在工业级流量上相较基线模型实现R@1提升+18.7%、R@3提升+7.1%、N@3提升+9.6%。我们还发现奖励设计至关重要:大模型常通过保留原始顺序或利用位置偏差“作弊”,因此条件可验证奖励是工业应用的核心组件。
原文摘要 · Abstract (English)
Industrial recommendation systems serve billions of users through a multi-stage funnel -- retrieval, early-stage ranking, and re-ranking -- where the final re-ranking step disproportionately shapes user engagement and downstream performance, particularly for carousel and grid display formats. Despite growing enthusiasm for Large Language Models (LLMs) in recommendation, three gaps hinder industrial adoption: (1) most efforts target retrieval and ranking, leaving re-ranking -- the stage closest to the final user experience -- largely underexplored; (2) LLMs are typically deployed zero-shot or via supervised fine-tuning, underutilizing the reasoning capabilities unlocked by reinforcement learning (RL) on verifiable rewards; (3) deployed catalogs index billions of items with non-semantic identifiers that lie outside any base-LLM vocabulary. We present GR2 (Generative Reasoning Re-Ranker), an end-to-end framework that combines (i) mid-training on semantic IDs produced by a tokenizer with >=99% uniqueness, (ii) reasoning-trace distilled from a stronger teacher via targeted prompting and rejection sampling, and (iii) RL with verifiable rewards purpose-built for re-ranking. To make GR2 resource-viable, we further (iv) introduce a context compressor that amortizes training cost, On-Policy Distillation (OPD) as a scalable alternative to SFT -- which we find collapses at industrial scale -- and reasoning distillation for low-latency serving. GR2 delivers +18.7% R@1, +7.1% R@3, and +9.6% N@3 over legacy baselines on industrial-scale traffic. We further find that reward design is critical in re-ranking: LLMs often hack rewards by preserving the incoming order or exploiting position bias, motivating conditional verifiable rewards as essential industrial components.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。