arXiv:2511.06937cs.IRcs.AI2025-11

用强化学习微调扩散推荐模型,提升推荐精度与效率。

Fine-Tuning Diffusion-Based Recommender Systems via Reinforcement Learning with Reward Function Optimization

  • 将去噪过程建模为马尔可夫决策过程,设计任务对齐的奖励函数。
  • 在序列推荐上性能提升最高达36.3%,且计算复杂度线性增长。
  • 适用于多种扩散推荐场景,适合追求高精度与高效微调的研究者。

扩散模型作为推荐系统的新范式,通过建模用户-物品交互的生成过程取得了顶尖性能。然而从头训练成本高昂,收敛后收益递减。为此,本文提出ReFiT框架,将基于强化学习的微调引入扩散推荐系统。与依赖外部奖励模型的现有方法不同,ReFiT采用任务对齐设计:将去噪轨迹建模为马尔可夫决策过程(MDP),并引入感知协同信号的奖励函数,直接反映推荐质量。通过紧密耦合MDP结构与该奖励信号,ReFiT使强化学习代理能利用高阶连接实现细粒度优化,避免了简单奖励设计中的噪声反馈问题。基于策略梯度优化,ReFiT最大化观测交互的精确对数似然,从而实现扩散推荐器的有效事后微调。在多个真实世界数据集上的实验表明,所提ReFiT框架(a)相比强基线有显著性能提升(序列推荐最高达36.3%),(b)具有线性复杂度(用户或物品数量),(c)在多种扩散推荐场景中具有良好泛化能力。源码与数据集已公开于https://anonymous.4open.science/r/ReFiT-4C60。

原文摘要 · Abstract (English)

Diffusion models recently emerged as a powerful paradigm for recommender systems, offering state-of-the-art performance by modeling the generative process of user-item interactions. However, training such models from scratch is both computationally expensive and yields diminishing returns once convergence is reached. To remedy these challenges, we propose ReFiT, a new framework that integrates Reinforcement learning (RL)-based Fine-Tuning into diffusion-based recommender systems. In contrast to prior RL approaches for diffusion models depending on external reward models, ReFiT adopts a task-aligned design: it formulates the denoising trajectory as a Markov decision process (MDP) and incorporates a collaborative signal-aware reward function that directly reflects recommendation quality. By tightly coupling the MDP structure with this reward signal, ReFiT empowers the RL agent to exploit high-order connectivity for fine-grained optimization, while avoiding the noisy or uninformative feedback common in naive reward designs. Leveraging policy gradient optimization, ReFiT maximizes exact log-likelihood of observed interactions, thereby enabling effective post hoc fine-tuning of diffusion recommenders. Comprehensive experiments on wide-ranging real-world datasets demonstrate that the proposed ReFiT framework (a) exhibits substantial performance gains over strong competitors (up to 36.3% on sequential recommendation), (b) demonstrates strong efficiency with linear complexity in the number of users or items, and (c) generalizes well across multiple diffusion-based recommendation scenarios. The source code and datasets are publicly available at https://anonymous.4open.science/r/ReFiT-4C60.

推荐系统扩散模型强化学习微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。