动态调整信任区域,让大模型强化学习更稳更准。
ETR: Outcome-Guided Elastic Trust Regions for Policy Optimization
- 根据信号质量动态调整优化约束,不再统一死板
- 在AIME和MATH上准确率更高,且熵值下降更慢
- 适合追求稳定推理能力的LLM强化学习研究者
基于可验证奖励的强化学习(RLVR)已成为激发大语言模型推理能力的重要范式,如OpenAI o1和DeepSeek-R1的成功所示。当前组相对策略优化(GRPO)是该领域主流算法,因其训练稳定且无需评判器。然而,我们指出GRPO存在结构性缺陷:对所有样本采用统一静态信任区域,隐含假设信号同质性,与结果驱动学习中优势幅度和方差高度异质的本质不符。这导致静态约束无法充分挖掘高质量信号,也无法有效抑制噪声,常引发快速熵崩溃。为此,我们提出弹性信任区域(ETR),一种随信号质量动态调整的机制。ETR通过双层弹性构建信号感知的优化景观:微观层面,依据优势大小缩放裁剪边界,加速高置信路径学习;宏观层面,利用组内方差隐式将更大更新预算分配给最优学习区任务。在AIME和MATH基准上的大量实验表明,ETR持续优于GRPO,不仅准确率更高,且有效缓解策略熵衰减,保障持续探索。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an important paradigm for unlocking reasoning capabilities in large language models, exemplified by the success of OpenAI o1 and DeepSeek-R1. Currently, Group Relative Policy Optimization (GRPO) stands as the dominant algorithm in this domain due to its stable training and critic-free efficiency. However, we argue that GRPO suffers from a structural limitation: it imposes a uniform, static trust region constraint across all samples. This design implicitly assumes signal homogeneity, a premise misaligned with the heterogeneous nature of outcome-driven learning, where advantage magnitudes and variances fluctuate significantly. Consequently, static constraints fail to fully exploit high-quality signals while insufficiently suppressing noise, often precipitating rapid entropy collapse. To address this, we propose \textbf{E}lastic \textbf{T}rust \textbf{R}egions (\textbf{ETR}), a dynamic mechanism that aligns optimization constraints with signal quality. ETR constructs a signal-aware landscape through dual-level elasticity: at the micro level, it scales clipping boundaries based on advantage magnitude to accelerate learning from high-confidence paths; at the macro level, it leverages group variance to implicitly allocate larger update budgets to tasks in the optimal learning zone. Extensive experiments on AIME and MATH benchmarks demonstrate that ETR consistently outperforms GRPO, achieving superior accuracy while effectively mitigating policy entropy degradation to ensure sustained exploration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。