arXiv:2601.18479cs.LG2026-01AAAI被引 3

通过状态转移定义相似状态,让动作更平滑。

Enhancing Control Policy Smoothness by Aligning Actions with Predictions from Preceding States

  • 用前一状态转移出的下一状态分布定义相似状态。
  • 在Gymnasium和Isaac-Lab中动作波动降低30%以上,性能提升。
  • 适合需要稳定控制的机器人、自动驾驶等真实场景。

深度强化学习在控制任务中表现强大,但其高频振荡难以应用于真实环境。以往基于损失的方法依赖启发式或人工定义的状态相似性来提升动作一致性,常无法准确反映系统动态。本文提出一种新方法,通过引入由前一状态转移产生的下一状态分布定义过渡相似状态,仅利用环境反馈与实际采集数据,更准确捕捉系统动态。基于此,我们提出动作平滑方法ASAP(Action Smoothing by Aligning Actions with Predictions from Preceding States),通过将当前动作与过渡相似状态中的动作对齐,并惩罚二阶差异来抑制高频振荡。在Gymnasium和Isaac-Lab环境中的实验表明,ASAP显著提升动作平滑度与策略性能,优于现有方法。

原文摘要 · Abstract (English)

Deep reinforcement learning has proven to be a powerful approach to solving control tasks, but its characteristic high-frequency oscillations make it difficult to apply in real-world environments. While prior methods have addressed action oscillations via architectural or loss-based methods, the latter typically depend on heuristic or synthetic definitions of state similarity to promote action consistency, which often fail to accurately reflect the underlying system dynamics. In this paper, we propose a novel loss-based method by introducing a transition-induced similar state. The transition-induced similar state is defined as the distribution of next states transitioned from the previous state. Since it utilizes only environmental feedback and actually collected data, it better captures system dynamics. Building upon this foundation, we introduce Action Smoothing by Aligning Actions with Predictions from Preceding States (ASAP), an action smoothing method that effectively mitigates action oscillations. ASAP enforces action smoothness by aligning the actions with those taken in transition-induced similar states and by penalizing second-order differences to suppress high-frequency oscillations. Experiments in Gymnasium and Isaac-Lab environments demonstrate that ASAP yields smoother control and improved policy performance over existing methods.

强化学习动作平滑控制优化机器人

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。