arXiv:2606.08480cs.LGcs.AI2026-06

针对推荐系统中奖励信号噪声问题,提出自适应选择性优化框架,提升生成式推荐效果。

Adaptive Loss Balancing for Noise-Robust GRPO in Generative Recommendation

论文配图:Adaptive Loss Balancing for Noise-Robust GRPO in Generative Recommendation
图 1 · 摘自论文原文
  • 根据策略不确定性和奖励判别力动态决定是否使用强化学习指导
  • 在电商数据集上使点击率指标提升至12.18%,幻觉率低于0.22%
  • 适合追求高可靠性与低幻觉的生成式推荐系统部署

强化学习为超越监督模仿的生成式推荐提供了新路径,依赖奖励信号引导策略优化。然而其有效性高度依赖奖励模型对样本评估的可信度。实际中,基于曝光偏差日志训练的生产级排序器存在样本相关误差,违背了这一假设。分层分析发现:当策略不确定且排序器能有效区分真实项与负样本时,奖励引导最为有益;其他情况下信号要么微弱,要么有害,暴露了统一应用RL的风险。为此,我们提出AdaGRPO框架,将奖励引导视为选择性准入而非统一施压。训练以监督负对数似然为基础,而GRPO目标由两个滚动诊断决定的二值门控控制:策略难度与奖励可判别性。不满足任一诊断的样本回归纯监督,保障稳定并抑制噪声梯度放大。在大规模电商数据集验证中,最佳中间检查点下HR@10从11.01%提升至12.18%,幻觉率低于0.22%;最终检查点仍保持稳健(HR@10 11.63%,幻觉率0.27%),优于固定NLL–GRPO混合方案。生产A/B测试显示点击率与停留时间显著提升,证实其实际价值。

原文摘要 · Abstract (English)

Reinforcement learning (RL) presents a promising avenue for enhancing generative recommendation beyond supervised imitation, leveraging reward signals to guide policy improvement. However, its efficacy is critically contingent on the trustworthiness of the reward model for the samples it evaluates. In practice, production rankers, the widely adopted reward models, are trained on exposure-biased logs, leading to sample-dependent inaccuracies that violate this assumption. Our stratified analysis uncovers a consistent pattern: reward guidance is most beneficial when the policy exhibits uncertainty and the ranker can effectively discriminate the ground-truth item from rollout negatives. On other samples, the reward signal is either negligible or detrimental, highlighting the risk of uniform RL application. To address such an issue, we introduce AdaGRPO, a novel framework that treats reward-guided optimization as selective admission rather than uniform pressure. Training is anchored in supervised negative log-likelihood, while the GRPO objective is gated by a binary, per-sample clip determined by two rollout diagnostics: policy-side difficulty and reward discriminability. Instances failing either diagnostic default to pure supervision, ensuring stability and mitigating the amplification of noisy gradients. We validate AdaGRPO on a large-scale e-commerce dataset. At the best intermediate checkpoint, it elevates HR@10 from 11.01% to 12.18% while constraining hallucination below 0.22%, and maintains robustness at the final checkpoint (HR@10 11.63%, hallucination 0.27%), outperforming fixed NLL--GRPO mixtures across the retrieval--validity frontier. In production A/B tests, AdaGRPO achieves statistically significant gains in click-through rate and dwell time, confirming its practical utility.

生成推荐强化学习噪声鲁棒自适应优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。