通过探索策略参数空间的未探区域，提升PPO等算法的性能。
Efficient On-Policy Reinforcement Learning via Exploration of Sparse Parameter Space
- 在每轮更新中系统性探测策略梯度附近的未探索区域。
- 无需增加梯度更新次数，在复杂连续控制任务中显著优于基线。
- 适用于希望提升现有PPO/TRPO性能的研究者或工程师。
策略梯度方法如近端策略优化（PPO）通常沿单一随机梯度方向更新，忽略了参数空间丰富的局部结构。已有研究指出，代理梯度与真实奖励景观相关性较差。基于此，我们可视化了单次迭代内策略检查点所覆盖的参数空间，发现高性能解常位于附近未探索区域。为此，我们提出ExploRLer，一个可插入PPO、TRPO等在线策略算法的通用流程，系统探测代理梯度更新的未探索邻域。无需增加梯度更新次数，ExploRLer在复杂连续控制环境中显著优于基线。结果表明，迭代级探索为增强在线强化学习提供了一种实用有效的方法，并为代理目标的局限性提供了新视角。
原文摘要 · Abstract (English)
Policy-gradient methods such as Proximal Policy Optimization (PPO) are typically updated along a single stochastic gradient direction, leaving the rich local structure of the parameter space unexplored. Previous work has shown that the surrogate gradient is often poorly correlated with the true reward landscape. Building on this insight, we visualize the parameter space spanned by policy checkpoints within an iteration and reveal that higher performing solutions often lie in nearby unexplored regions. To exploit this opportunity, we introduce ExploRLer, a pluggable pipeline that seamlessly integrates with on-policy algorithms such as PPO and TRPO, systematically probing the unexplored neighborhoods of surrogate on-policy gradient updates. Without increasing the number of gradient updates, ExploRLer achieves significant improvements over baselines in complex continuous control environments. Our results demonstrate that iteration-level exploration provides a practical and effective way to strengthen on-policy reinforcement learning and offer a fresh perspective on the limitations of the surrogate objective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。