提升强化学习策略平滑性，减少硬件部署时的高频振荡。
Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies
- 通过损失正则化与网络结构设计两类方法提升策略平滑性。
- 混合方法使控制平滑度提升26.8%，性能最差仅下降2.8%。
- 在仿真与真实机器人上均验证有效，适合实际硬件部署场景。
强化学习策略易产生高频振荡，尤其在真实世界硬件部署时尤为不利。本文系统梳理并对比了文献中缓解深度强化学习高频振荡的方法，将其分为损失正则化与架构设计两大类。核心思想是促使策略学习一个平滑映射，使输入空间中邻近状态对应输出动作也趋于相近。我们在Gymnasium经典环境、复杂操作任务及三个包含真实硬件部署与评估的机器人运动任务上进行了基准测试。此外，提出结合两类方法的混合策略。结果表明，最优混合方法显著优于其他方法，相比基线控制平滑度提升26.8%，最坏情况性能下降仅2.8%。
原文摘要 · Abstract (English)
Reinforcement learning (RL) policies are prone to high-frequency oscillations, especially undesirable when deploying to hardware in the real-world. In this paper, we identify, categorize, and compare methods from the literature that aim to mitigate high-frequency oscillations in deep RL. We define two broad classes: loss regularization and architectural methods. At their core, these methods incentivize learning a smooth mapping, such that nearby states in the input space produce nearby actions in the output space. We present benchmarks in terms of policy performance and control smoothness on traditional RL environments from the Gymnasium and a complex manipulation task, as well as three robotics locomotion tasks that include deployment and evaluation with real-world hardware. Finally, we also propose hybrid methods that combine elements from both loss regularization and architectural methods. We find that the best-performing hybrid outperforms other methods, and improves control smoothness by 26.8% over the baseline, with a worst-case performance degradation of just 2.8%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。