提出EnhancedRL算法,用用户和物品特征融合提升推荐系统多任务融合效果。
EnhancedRL: An Enhanced-State Reinforcement Learning Algorithm for Multi-Task Fusion in Recommender Systems
- 引入增强状态,融合用户、物品及额外特征作为强化学习输入
- 在真实工业系统中实现用户有效消费+3.84%、停留时长+0.58%
- 首个支持用户-物品对级优化的推荐系统强化学习方法,适合大规模部署
推荐系统中的多任务融合(MTF)负责将多任务学习输出的多个评分合并为单一评分,最终决定推荐结果。近年来,强化学习(RL)被用于MTF,以在推荐会话中最大化长期用户满意度。然而,现有RL-MTF方法仅依赖用户特征与统计特征作为状态,在用户层面生成动作,无法利用物品特征及其他有价值信息,导致性能受限。突破这一瓶颈需革新建模范式,但迄今尚无研究解决该问题。为此,本文提出EnhancedRL,一种创新的强化学习算法。不同于以往方法,EnhancedRL采用增强状态,包含用户特征、物品特征及其他有价值信息。同时,设计定制化演员-评论家框架,包括重构的演员与评论家模块及新型学习流程,可在推荐会话内实现用户-物品对级别的长期奖励优化。在工业级推荐系统上开展的离线与在线实验表明,EnhancedRL显著优于其他方法,带来用户有效消费+3.84%和用户停留时长+0.58%的提升。据我们所知,EnhancedRL是首个解决该挑战的工作,自2023年9月14日起已在大规模推荐系统中全面部署并取得显著成效。
原文摘要 · Abstract (English)
As a key stage of Recommender Systems (RSs), Multi-Task Fusion (MTF) is responsible for merging multiple scores output by Multi-Task Learning (MTL) into a single score, finally determining the recommendation results. Recently, Reinforcement Learning (RL) has been applied to MTF to maximize long-term user satisfaction within a recommendation session. However, due to limitations in modeling paradigm, all existing RL algorithms for MTF can only utilize user features and statistical features as the state to generate actions at the user level, but unable to leverage item features and other valuable features, which leads to suboptimal performance. Overcoming this problem requires a breakthrough in the existing modeling paradigm, yet, to date, no prior work has addressed it. To tackle this challenge, we propose EnhancedRL, an innovative RL algorithm. Unlike existing RL-MTF methods, EnhancedRL takes the enhanced state as input, incorporating not only user features but also item features and other valuable information. Furthermore, it introduces a tailored actor-critic framework - including redesigned actor and critics and a novel learning procedure - to optimize long-term rewards at the user-item pair level within a recommendation session. Extensive offline and online experiments are conducted in an industrial RS and the results demonstrate that EnhancedRL outperforms other methods remarkably, achieving a +3.84% increase in user valid consumption and a +0.58% increase in user duration time. To the best of our knowledge, EnhancedRL is the first work to address this challenge, and it has been fully deployed in a large-scale RS since September 14, 2023, yielding significant improvements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。