用大模型+强化学习,让推荐系统更聪明、更全面。
RecLLM-R1: A Two-Stage Training Paradigm with Reinforcement Learning and Chain-of-Thought v1
- 分两阶段训练:先微调,再用强化学习优化决策过程。
- 在真实社交平台数据上,准确率、多样性均显著优于基线方法。
- 适合需要兼顾精准推荐与业务目标的复杂场景应用。
传统推荐系统常面临信息茧房、外部知识利用不足及模型优化与业务策略脱节等问题。为此,本文提出RecLLM-R1框架,基于大语言模型(LLM)并借鉴DeepSeek R1方法。首先通过精心设计的数据构建流程,将用户画像、历史行为和多维度商品属性转化为大模型可理解的自然语言提示。随后采用两阶段训练:第一阶段通过监督微调(SFT)赋予模型基础推荐能力;第二阶段使用组相对策略优化(GRPO)结合思维链(CoT)机制,通过灵活定义的奖励函数引导模型进行多步推理和整体决策,同时优化推荐准确性、多样性等定制化业务目标。在某大型社交平台的真实用户行为数据集上的实验表明,RecLLM-R1在准确率、多样性、新颖性等多项指标上显著优于现有基线方法,有效缓解了信息茧房问题,为复杂业务目标下推荐模型与策略的协同优化提供了可行路径。
原文摘要 · Abstract (English)
Traditional recommendation systems often grapple with "filter bubbles", underutilization of external knowledge, and a disconnect between model optimization and business policy iteration. To address these limitations, this paper introduces RecLLM-R1, a novel recommendation framework leveraging Large Language Models (LLMs) and drawing inspiration from the DeepSeek R1 methodology. The framework initiates by transforming user profiles, historical interactions, and multi-faceted item attributes into LLM-interpretable natural language prompts through a carefully engineered data construction process. Subsequently, a two-stage training paradigm is employed: the initial stage involves Supervised Fine-Tuning (SFT) to imbue the LLM with fundamental recommendation capabilities. The subsequent stage utilizes Group Relative Policy Optimization (GRPO), a reinforcement learning technique, augmented with a Chain-of-Thought (CoT) mechanism. This stage guides the model through multi-step reasoning and holistic decision-making via a flexibly defined reward function, aiming to concurrently optimize recommendation accuracy, diversity, and other bespoke business objectives. Empirical evaluations on a real-world user behavior dataset from a large-scale social media platform demonstrate that RecLLM-R1 significantly surpasses existing baseline methods across a spectrum of evaluation metrics, including accuracy, diversity, and novelty. It effectively mitigates the filter bubble effect and presents a promising avenue for the integrated optimization of recommendation models and policies under intricate business goals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。