arXiv:2603.10279cs.LG2026-03

用指数奖励加权微调提升生成推荐系统鲁棒性,效果优于强化学习。

Robust Post-Training for Generative Recommenders: Why Exponential Reward-Weighted SFT Outperforms RLHF

  • 直接基于观测奖励优化,不依赖学习的奖励模型。
  • 在三个开源和一个私有数据集上均显著优于RLHF基线。
  • 温度参数λ可量化调控鲁棒性与性能平衡,适合工业部署。

通过后训练对齐生成推荐系统与用户偏好,是缩小下一物品预测与实际推荐质量差距的关键。现有方法在生产级系统中表现不佳:强化学习人类反馈(RLHF)易受噪声反馈影响导致奖励作弊,离线强化学习需倾向得分但不可得,在线交互又不现实。本文提出指数奖励加权监督微调(SFT),权重为 $w = \ ext{exp}(r/λ)$,具备理论与实证基础。该方法直接优化可观测奖励,无需查询学习的奖励模型,免于奖励作弊,无需倾向得分,且完全离线。首次证明在噪声奖励下该方法具有策略改进保证,差距仅随目录规模对数增长,即使在大规模商品库中仍具信息量。关键发现:温度 $λ$ 显式控制鲁棒性与性能的权衡,为从业者提供一个可解释、有理论依据的正则化超参。在三个开源及一个私有数据集上,对比四种基线的实验验证了其简单、可扩展且持续领先的效果。

原文摘要 · Abstract (English)

Aligning generative recommender systems to user preferences via post-training is critical for closing the gap between next-item prediction and actual recommendation quality. Existing post-training methods are ill-suited for production-scale systems: RLHF methods reward hack due to noisy user feedback and unreliable reward models, offline RL alternatives require propensity scores that are unavailable, and online interaction is infeasible. We identify exponential reward-weighted SFT with weights $w = \exp(r/λ)$ as uniquely suited to this setting, and provide the theoretical and empirical foundations that explain why. By optimizing directly on observed rewards without querying a learned reward model, the method is immune to reward hacking, requires no propensity scores, and is fully offline. We prove the first policy improvement guarantees for this setting under noisy rewards, showing that the gap scales only logarithmically with catalog size and remains informative even for large item catalogs. Crucially, we show that temperature $λ$ explicitly and quantifiably controls the robustness-improvement tradeoff, providing practitioners with a single interpretable regularization hyperparameter with theoretical grounding. Experiments on three open-source and one proprietary dataset against four baselines confirm that exponential reward weighting is simple, scalable, and consistently outperforms RLHF-based alternatives.

生成推荐SFT鲁棒性奖励加权

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。