arXiv:2510.23077cs.IRcs.AI2025-10NeurIPS被引 12

用强化学习让推荐模型自主思考,提升预测准确率。

Think before Recommendation: Autonomous Reasoning-enhanced Recommender

  • 用结构化提示引导模型分步分析用户兴趣与物品特征。
  • 通过规则奖励机制优化推理路径,实现端到端训练。
  • 适合想提升推荐系统智能推理能力的研究者和工程师。

推荐系统的核心任务是从历史用户-物品交互中学习用户偏好。随着大语言模型(LLM)的快速发展,近期研究尝试利用其推理能力增强评分预测。然而,现有基于蒸馏的方法存在教师模型推荐能力不足、监督成本高且静态、推理能力浅层迁移等问题。为此,本文提出RecZero,一种基于强化学习(RL)的推荐范式,摒弃传统的多模型多阶段蒸馏方法,仅通过纯强化学习训练单个LLM,使其自主发展评分预测的推理能力。RecZero包含两个关键组件:(1) “先思考再推荐”提示构建,采用结构化推理模板引导模型逐步分析用户兴趣、物品特征及用户-物品匹配度;(2) 基于规则的奖励建模,使用组相对策略优化(GRPO)计算推理轨迹奖励并优化LLM。此外,论文还探索了混合范式RecOne,结合监督微调与强化学习,以冷启动推理样本初始化模型,并进一步通过强化学习优化。实验结果表明,RecZero和RecOne在多个基准数据集上显著优于现有基线方法,验证了强化学习范式在实现自主推理增强推荐系统中的优越性。

原文摘要 · Abstract (English)

The core task of recommender systems is to learn user preferences from historical user-item interactions. With the rapid development of large language models (LLMs), recent research has explored leveraging the reasoning capabilities of LLMs to enhance rating prediction tasks. However, existing distillation-based methods suffer from limitations such as the teacher model's insufficient recommendation capability, costly and static supervision, and superficial transfer of reasoning ability. To address these issues, this paper proposes RecZero, a reinforcement learning (RL)-based recommendation paradigm that abandons the traditional multi-model and multi-stage distillation approach. Instead, RecZero trains a single LLM through pure RL to autonomously develop reasoning capabilities for rating prediction. RecZero consists of two key components: (1) "Think-before-Recommendation" prompt construction, which employs a structured reasoning template to guide the model in step-wise analysis of user interests, item features, and user-item compatibility; and (2) rule-based reward modeling, which adopts group relative policy optimization (GRPO) to compute rewards for reasoning trajectories and optimize the LLM. Additionally, the paper explores a hybrid paradigm, RecOne, which combines supervised fine-tuning with RL, initializing the model with cold-start reasoning samples and further optimizing it with RL. Experimental results demonstrate that RecZero and RecOne significantly outperform existing baseline methods on multiple benchmark datasets, validating the superiority of the RL paradigm in achieving autonomous reasoning-enhanced recommender systems.

推荐系统强化学习大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。