提出SAFE算法，解决RLHF中PPO的不稳定问题
SAFE: Stable Alignment Finetuning with Entropy-Aware Predictive Control for Reinforcement Learning from Human Feedback (RLHF)
- 用双软最小评论器和熵感知调节实现更稳定的策略优化
- 30亿参数模型上比PPO高5.15%平均奖励，无奖励崩溃
- 适合需要稳定部署的生成式AI系统，代码开源
近期文献将近端策略优化（PPO）视为强化学习从人类反馈（RLHF）中强化学习部分的标准方法。尽管PPO在实践中表现良好，但其动机具有启发性，对语言模型RLHF中使用的KL散度约束处理方式较为随意，且存在奖励震荡、熵塌陷、价值函数漂移和策略突变等问题，需频繁重启和大量超参数调优。本文提出一种新的纯在线策略演员-评论家强化学习方法，用于语言模型RLHF场景。SAFE（Stable Alignment Finetuning with Entropy-aware control）结合双软最小评论器进行悲观价值估计，以及包含熵门控KL调节和PID控制自适应阈值的多层稳定框架。与标准PPO的对称KL惩罚不同，SAFE区分高熵探索与低熵模式坍缩，并根据奖励速度动态调整惩罚。在30亿参数模型上的实验表明，SAFE相比PPO提升5.15%的平均奖励（0.725 vs 0.689），几乎无奖励崩溃，且KL控制更优。该方法计算开销极小，提供可解释、抗崩溃的RLHF框架，在保持激进学习速度的同时确保长期优化稳定性，适用于生产部署。代码已公开于https://github.com/ryyzn9/SAFE。
原文摘要 · Abstract (English)
Proximal Policy Optimization (PPO) has been positioned by recent literature as the canonical method for the RL part of Reinforcement Learning from Human Feedback (RLHF). PPO performs well empirically but has a heuristic motivation and handles the KL-divergence constraint used in LM-RLHF in an ad-hoc manner and suffers form reward oscillations, entropy collapse, value function drift, and sudden policy divergence that require frequent restarts and extensive hyperparameter tuning. In this paper, we develop a new pure on policy actor-critic RL method for the LM-RLHF setting. We present SAFE (Stable Alignment Finetuning with Entropy-aware control),a novel RLHF algorithm that combines a Double Soft-Min Critic for pessimistic value estimation with a new multi-layer stabilization framework combining entropy-gated KL regulation, and PID-controlled adaptive thresholds. Unlike standard PPO's symmetric KL penalties, SAFE distinguishes high-entropy exploration from low-entropy mode collapse and adjusts penalties dynamically based on reward velocity. Experiments on a 3B parameter model show SAFE achieves +5.15\% training-average reward than PPO (0.725 vs 0.689), negligible reward crashes, and superior KL control than ppo . Our method adds minimal computational overhead and provides an interpretable, crash-resistant RLHF framework that maintains aggressive learning speed while ensuring stable long-horizon optimization suitable for production deployment. Code is available at https://github.com/ryyzn9/SAFE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。