提出新型策略优化方法ANO，解决强化学习中梯度不稳定问题
ANO: A Principled Approach to Robust Policy Optimization
- 用红降梯度机制替代PPO的硬截断，实现梯度抑制与平滑恢复
- 在连续和离散控制任务中抗极端学习率，防止策略崩溃
- 在大模型对齐中避免KL爆炸，性能超越PPO/SPO等主流方法
近端策略优化（PPO）虽在强化学习和大模型对齐中占据主导地位，但依赖的‘硬截断’机制会丢弃有价值的梯度。而无约束方法如SPO在遭遇极端异常值时易引发剧烈不稳定性与策略崩溃。为此，本文提出一种原则性设计空间，证明稳健估计器必须内生抑制异常值并保持平滑恢复力。基于此几何原理，推导出锚定邻域优化（ANO），以红降梯度机制无缝替代硬截断。大量实验表明，ANO在多种场景下均具优越性：在连续控制（MuJoCo）与离散控制（Atari）任务中，即使使用高达 $1 imes 10^{-3}$ 的激进学习率，仍能有效防止策略崩溃；在大模型对齐（RLHF）中，明确消除无约束方法固有的灾难性KL发散，头对头对比中胜过PPO、SPO与GRPO。
原文摘要 · Abstract (English)
Proximal Policy Optimization (PPO) dominates reinforcement learning and LLM alignment but relies on a "hard clipping" mechanism that discards valuable gradients. Conversely, unconstrained methods like SPO expose the optimization to unbounded updates, causing severe instability and policy collapse during extreme outlier encounters. To resolve this dilemma, we introduce a principled design space for policy optimization, demonstrating that a robust estimator must inherently suppress outliers while maintaining a smooth restoration force. Guided by these geometric principles, we derive Anchored Neighborhood Optimization (ANO), a novel method that seamlessly replaces hard clipping with a redescending gradient mechanism. Extensive evaluations demonstrate ANO's empirical superiority across diverse domains. In continuous (MuJoCo) and discrete (Atari) control, ANO establishes a robust state-of-the-art, uniquely preventing policy collapse even under highly aggressive learning rates ($1 \times 10^{-3}$). Furthermore, in LLM alignment (RLHF), ANO explicitly eliminates the catastrophic KL divergence explosion inherent to unconstrained methods, dominating PPO, SPO, and GRPO in head-to-head win rates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。