用强化学习提升大模型推荐效率,解决推理慢、训练不稳问题
Towards Sample-Efficient and Stable Reinforcement Learning for LLM-based Recommendation
- 摒弃传统思维链,直接用强化学习探索物品空间
- 在三个真实数据集上显著超越基线模型,推荐效果更优
- 适合关注大模型推荐系统稳定性和样本效率的研究者
尽管长链式思维(Long CoT)在大语言模型中展现潜力,但其在序列推荐领域存在根本性不适配:推理延迟过高,且用户行为数据缺乏显式的认知推理模式。为此,我们提出转向强化学习(RL),直接利用其探索物品空间的能力。然而,直接应用RL面临样本效率低和训练不稳定的问题。为此,我们提出RISER框架,将不可学习的推荐轨迹转化为有效的成对偏好数据,并引入防冗余回放与逐标记更新幅度约束策略以保证稳定性。在三个真实数据集上的实验表明,RISER显著优于现有基线,为增强型大模型推荐建立了可靠范式。
原文摘要 · Abstract (English)
While Long Chain-of-Thought (Long CoT) reasoning has shown promise in Large Language Models (LLMs), its adoption for enhancing recommendation quality is growing rapidly. In this work, we critically examine this trend and argue that Long CoT is inherently ill-suited for the sequential recommendation domain. We attribute this misalignment to two primary factors: excessive inference latency and the lack of explicit cognitive reasoning patterns in user behavioral data. Driven by these observations, we propose pivoting away from the CoT structure to directly leverage its underlying mechanism: Reinforcement Learning (RL), to explore the item space. However, applying RL directly faces significant obstacles, notably low sample efficiency-where most actions fail to provide learning signals-and training instability. To overcome these limitations, we propose RISER, a novel Reinforced Item Space Exploration framework for Recommendation. RISER is designed to transform non-learnable trajectories into effective pairwise preference data for optimization. Furthermore, it incorporates specific strategies to ensure stability, including the prevention of redundant rollouts and the constraint of token-level update magnitudes. Extensive experiments on three real-world datasets show that RISER significantly outperforms competitive baselines, establishing a robust paradigm for RL-enhanced LLM recommendation. Our code will be available at https://anonymous.4open.science/r/RISER/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。