用新算法让大模型推荐更准更稳定,避免乱推荐和排序失效。
Rank-GRPO: Training LLM-based Conversational Recommender Systems with Reinforcement Learning
- 分两阶段训练:先用黑盒模型生成高质量对话数据,再用新强化学习算法优化推荐排序。
- 在Reddit-v2数据集上,召回率和NDCG均优于传统方法,收敛更快。
- 适合想用大模型做对话推荐的开发者,尤其关注推荐结果质量与稳定性。
大型语言模型正在改变推荐系统范式,使用户可通过对话表达偏好并获得推荐。然而,将预训练大模型对齐推荐任务仍具挑战:模型常生成不在目录中的项目、违反输出格式要求,且推荐列表末端的排序质量显著下降。为此,我们提出ConvRec-R1,一个用于端到端训练基于大模型的对话推荐系统的两阶段框架。第一阶段,通过重映射-反思-调整管道构建行为克隆数据集,利用强大黑盒大模型生成高质量、符合目录约束的示范数据,以初始化强化学习训练。第二阶段,提出Rank-GRPO,一种针对具有排序输出任务的组相对策略优化(GRPO)的合理扩展。Rank-GRPO以推荐列表中的每个排名位置为优化单元,而非单个词元(过细)或整个序列(过粗),重新定义奖励机制以消除非因果信用分配,并引入基于排名内词元概率几何平均的等级重要性比率,稳定策略更新。在公开的Reddit-v2数据集上的实验表明,ConvRec-R1收敛更快,且在召回率和NDCG指标上优于基于GRPO的基线方法。代码与数据集已开源于https://github.com/yaochenzhu/Rank-GRPO。
原文摘要 · Abstract (English)
Large language models (LLMs) are reshaping the recommender system paradigm by enabling users to express preferences and receive recommendations through conversations. Yet, aligning LLMs to the recommendation task remains challenging: pretrained LLMs often generate out-of-catalog items, violate required output formats, and their ranking quality degrades sharply toward the end of the generated list. To this end, we propose ConvRec-R1, a two-stage framework for end-to-end training of LLM-based conversational recommender systems. In Stage 1, we construct a behavioral-cloning dataset with a Remap-Reflect-Adjust pipeline, which produces high-quality, catalog-grounded demonstrations from powerful blackbox LLMs to warm-start the RL training. In Stage 2, we propose Rank-GRPO, a principled extension of group relative policy optimization (GRPO) tailored to tasks with rank-style outputs. Rank-GRPO treats each rank in the recommendation list as the unit instead of token (too fine-grained) or sequence (too coarse), redefining rewards to remove non-causal credit assignment and introducing a rank-level importance ratio based on the geometric mean of rank-wise token probabilities to stabilize policy updates. Experiments on the public Reddit-v2 dataset show that ConvRec-R1 converges faster and achieves higher Recall and NDCG than GRPO-style baselines. Code and datasets are released at https://github.com/yaochenzhu/Rank-GRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。