arXiv:2506.19235cs.AI2025-06被引 8

用大模型+强化学习,让推荐系统更聪明、更全面。

RecLLM-R1: A Two-Stage Training Paradigm with Reinforcement Learning and Chain-of-Thought v1

  • 分两阶段训练:先微调,再用强化学习优化决策过程。
  • 在真实社交平台数据上,准确率、多样性均显著优于基线方法。
  • 适合需要兼顾精准推荐与业务目标的复杂场景应用。

传统推荐系统常面临信息茧房、外部知识利用不足及模型优化与业务策略脱节等问题。为此,本文提出RecLLM-R1框架,基于大语言模型(LLM)并借鉴DeepSeek R1方法。首先通过精心设计的数据构建流程,将用户画像、历史行为和多维度商品属性转化为大模型可理解的自然语言提示。随后采用两阶段训练:第一阶段通过监督微调(SFT)赋予模型基础推荐能力;第二阶段使用组相对策略优化(GRPO)结合思维链(CoT)机制,通过灵活定义的奖励函数引导模型进行多步推理和整体决策,同时优化推荐准确性、多样性等定制化业务目标。在某大型社交平台的真实用户行为数据集上的实验表明,RecLLM-R1在准确率、多样性、新颖性等多项指标上显著优于现有基线方法,有效缓解了信息茧房问题,为复杂业务目标下推荐模型与策略的协同优化提供了可行路径。

原文摘要 · Abstract (English)

Traditional recommendation systems often grapple with "filter bubbles", underutilization of external knowledge, and a disconnect between model optimization and business policy iteration. To address these limitations, this paper introduces RecLLM-R1, a novel recommendation framework leveraging Large Language Models (LLMs) and drawing inspiration from the DeepSeek R1 methodology. The framework initiates by transforming user profiles, historical interactions, and multi-faceted item attributes into LLM-interpretable natural language prompts through a carefully engineered data construction process. Subsequently, a two-stage training paradigm is employed: the initial stage involves Supervised Fine-Tuning (SFT) to imbue the LLM with fundamental recommendation capabilities. The subsequent stage utilizes Group Relative Policy Optimization (GRPO), a reinforcement learning technique, augmented with a Chain-of-Thought (CoT) mechanism. This stage guides the model through multi-step reasoning and holistic decision-making via a flexibly defined reward function, aiming to concurrently optimize recommendation accuracy, diversity, and other bespoke business objectives. Empirical evaluations on a real-world user behavior dataset from a large-scale social media platform demonstrate that RecLLM-R1 significantly surpasses existing baseline methods across a spectrum of evaluation metrics, including accuracy, diversity, and novelty. It effectively mitigates the filter bubble effect and presents a promising avenue for the integrated optimization of recommendation models and policies under intricate business goals.

推荐系统大模型强化学习思维链

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。