提出新算法解决大模型强化学习探索崩溃问题
Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization
- 基于黎曼几何设计等距策略更新，纠正原有方法的度量错误
- 在7个竞赛级基准上提升显著，最高比GRPO高出60%
- 适合研究大模型强化学习与策略优化的学者和工程师
强化学习已成为提升大模型推理能力的主要范式。然而，以PPO-Clip为代表的算法存在探索崩溃的固有缺陷。本文揭示其根本原因：使用欧氏度量衡量策略差异，与策略黎曼流形的内在几何不一致，导致低概率区域更新过保守、高概率区域更新过激，最终引发探索崩溃。为此，提出黎曼等距策略优化（RIPO），保证在黎曼流形上的等距更新，有效平衡探索与利用。进一步证明RIPO具有优良的偏差-方差权衡，稳定优化过程。大量实验表明，RIPO在7个竞赛级基准上显著优于现有算法，最高较GRPO提升60%。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become a dominant paradigm for enhancing LLMs' reasoning capabilities. However, RL algorithms with PPO-Clip are inherently limited by exploration collapse. Subsequent works remain primarily heuristic and fail to identify the essential cause of PPO-Clip's failure. This work reveals the fundamental flaw of PPO-Clip: it implicitly measures policy discrepancy using Euclidean metric, which is theoretically inconsistent with the intrinsic geometry on the policy Riemannian manifold. This geometric mismatch results in overly conservative updates in low-probability regions while aggressive in high-probability regions, ultimately collapsing exploration. To correct this geometric flaw, we propose Riemannian Isometric Policy Optimization (RIPO), which guarantees isometric policy updates on the Riemannian manifold, effectively balancing exploration and exploitation. We further show that RIPO achieves a favorable bias-variance trade-off, which stabilizes optimization. Extensive experiments demonstrate that RIPO significantly surpasses existing LLM RL algorithms across seven competition-level benchmarks (up to 60% improvement over GRPO on AIME24).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。