通过博弈论设计奖励模型，让大模型更符合用户偏好。
Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective
- 将奖励优化建模为斯塔克尔伯格博弈，动态调整奖励函数。
- 在推理时对齐中提升平均奖励，胜平率超66%。
- 兼容现有方法，只需少量计算开销，适合实际部署。
现有对齐方法直接使用从用户偏好数据中学习到的奖励模型来优化大语言模型策略，同时施加相对于基础策略的KL正则化。这种做法在最大化用户效用方面表现不佳，因为KL正则化可能导致大模型继承与用户偏好相冲突的基础策略偏差。虽然放大偏好输出的奖励可缓解偏差问题，但会增加奖励欺骗风险。这一权衡促使我们研究在KL正则化下最优奖励模型的设计。本文将该问题形式化为斯塔克尔伯格博弈，并证明一种简单的奖励塑造方案能有效逼近最优奖励模型。我们在推理时对齐设置中进行了实证评估，结果表明该方法可无缝集成到现有对齐方法中，且开销极小。所提方法在不同评估设置下平均奖励持续提升，胜平率超过66%。
原文摘要 · Abstract (English)
Existing alignment methods directly use the reward model learned from user preference data to optimize an LLM policy, subject to KL regularization with respect to the base policy. This practice is suboptimal for maximizing user's utility because the KL regularization may cause the LLM to inherit the bias in the base policy that conflicts with user preferences. While amplifying rewards for preferred outputs can mitigate this bias, it also increases the risk of reward hacking. This tradeoff motivates the problem of optimally designing reward models under KL regularization. We formalize this reward model optimization problem as a Stackelberg game, and show that a simple reward shaping scheme can effectively approximate the optimal reward model. We empirically evaluate our method in inference-time alignment settings and demonstrate that it integrates seamlessly into existing alignment methods with minimal overhead. Our method consistently improves average reward and achieves win-tie rates exceeding 66% against all baselines, averaged across evaluation settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。