统一处理奖励欺骗与优化稳定，提升强化学习对齐效果。
Unifying Stable Optimization and Reference Regularization in RLHF
- 用加权监督微调损失统一正则化目标
- 在多个基准上同时提升对齐性能与稳定性
- 适合关注对齐训练稳定性的研究者
基于人类反馈的强化学习（RLHF）显著提升了模型对齐能力，但仍面临两大核心挑战：奖励欺骗和优化不稳定。现有方法分别通过对抗监督微调模型（π₀）的KL散度惩罚来缓解奖励欺骗，以及对当前策略（πₜ）进行策略比例裁剪以保证优化稳定。然而，同时向π₀和πₜ正则化所引发的隐含权衡尚未被充分探索。本文提出一种统一正则化方法，显式平衡防止奖励欺骗与保持策略更新稳定的目标。所提出的原理清晰的对齐目标，生成具有更优权衡的加权监督微调损失，在对齐效果和实现复杂度上均表现更优。在多种基准上的广泛实验表明，该方法持续优于RLHF与在线偏好学习方法，实现了更高的对齐性能与更强的稳定性。
原文摘要 · Abstract (English)
Reinforcement Learning from Human Feedback (RLHF) has advanced alignment capabilities significantly but remains hindered by two core challenges: \textbf{reward hacking} and \textbf{stable optimization}. Current solutions independently address these issues through separate regularization strategies, specifically a KL-divergence penalty against a supervised fine-tuned model ($π_0$) to mitigate reward hacking, and policy ratio clipping towards the current policy ($π_t$) to promote stable alignment. However, the implicit trade-off arising from simultaneously regularizing towards both $π_0$ and $π_t$ remains under-explored. In this paper, we introduce a unified regularization approach that explicitly balances the objectives of preventing reward hacking and maintaining stable policy updates. Our simple yet principled alignment objective yields a weighted supervised fine-tuning loss with a superior trade-off, which demonstrably improves both alignment results and implementation complexity. Extensive experiments across diverse benchmarks validate that our method consistently outperforms RLHF and online preference learning methods, achieving enhanced alignment performance and stability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。