用大模型提升推荐系统探索能力，缓解线上部署风险
Large Language Model driven Policy Exploration for Recommender Systems
- 用大模型分析用户状态生成偏好，预训练推荐策略
- 在线版本A-iALP在三个环境提升推荐性能，尤其改善长期收益
- 适合关注长时效果、需安全上线的推荐系统研究者
推荐系统近年引入强化学习（RL），将推荐建模为马尔可夫决策过程（MDP）。然而，基于静态用户数据训练的离线RL策略在动态线上环境中易受分布偏移影响。过度关注短期相关物品会抑制探索，导致次优推荐并损害长期用户收益。在线RL部署也面临暴露于未训练或不稳定策略的风险。大语言模型（LLM）可通过模拟用户目标与偏好，在离线阶段预训练策略以增强线上初始推荐表现。为此，我们提出交互增强型学习策略iALP，利用LLM从用户状态中提炼物品偏好，基于反馈学习奖励，并通过演员-评论家框架更新RL策略。为进一步实现在线部署，我们设计自适应变体A-iALP，包含简单微调策略（A-iALP$_{ft}$）与抗策略退化方法（A-iALP$_{ap}$）。在三个模拟环境中的实验表明，A-iALP显著提升推荐性能。
原文摘要 · Abstract (English)
Recent advancements in Recommender Systems (RS) have incorporated Reinforcement Learning (RL), framing the recommendation as a Markov Decision Process (MDP). However, offline RL policies trained on static user data are vulnerable to distribution shift when deployed in dynamic online environments. Additionally, excessive focus on exploiting short-term relevant items can hinder exploration, leading to suboptimal recommendations and negatively impacting long-term user gains. Online RL-based RS also face challenges in production deployment, due to the risks of exposing users to untrained or unstable policies. Large Language Models (LLMs) offer a promising solution to mimic user objectives and preferences for pre-training policies offline to enhance the initial recommendations in online settings. Effectively managing distribution shift and balancing exploration are crucial for improving RL-based RS, especially when leveraging LLM-based pre-training. To address these challenges, we propose an Interaction-Augmented Learned Policy (iALP) that utilizes user preferences distilled from an LLM. Our approach involves prompting the LLM with user states to extract item preferences, learning rewards based on feedback, and updating the RL policy using an actor-critic framework. Furthermore, to deploy iALP in an online scenario, we introduce an adaptive variant, A-iALP, that implements a simple fine-tuning strategy (A-iALP$_{ft}$), and an adaptive approach (A-iALP$_{ap}$) designed to mitigate issues with compromised policies and limited exploration. Experiments across three simulated environments demonstrate that A-iALP introduces substantial performance improvements
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。