通过梯度惩罚增强潜空间动态平滑性,提升强化学习采样效率
Dreaming Smoothly and Sample Efficiently with Gradient Penalized Latent Dynamics

- 在DreamerV3基础上加入行级雅可比惩罚,强制潜变量转移动态局部平滑
- 在高复杂度运动任务中,样本效率显著提升,四足任务更早达到高回报
- 方法简单有效,适合连续控制类强化学习任务的改进
基于模型的强化学习通过学习世界模型提升采样效率。然而,现有潜空间世界模型(如DreamerV3)未显式约束其转移动态的局部平滑性,忽略了这一对动态学习有益的归纳偏置。本文提出GPLD,一种针对DreamerV3的梯度惩罚潜动态正则化方法,通过对后验潜变量分布施加行级雅可比惩罚,促进局部平滑的转移学习。该惩罚可解释为离散嵌入状态马尔可夫决策过程中的有限差分平滑的连续潜空间类比,并通过Hutchinson风格的随机探测高效估计。在DeepMind Control本体感知任务上,GPLD提升了综合采样效率,尤其在更高复杂度的运动环境中表现突出。在更具挑战性的四足任务中,GPLD更早实现高回报行为,并在更长时域内表现出更稳定的后期学习。显式局部平滑正则化是提升连续控制环境潜空间世界模型的有效且简洁方法。代码见github.com/romils9/gpld-mbrl。
原文摘要 · Abstract (English)
Model-based reinforcement learning improves sample efficiency by learning a world model. However, existing latent world models such as DreamerV3 do not explicitly enforce local smoothness in their learned transition dynamics, leaving a useful inductive bias for transition dynamics learning unexploited. We propose GPLD, a gradient-penalized latent dynamics regularizer for DreamerV3 that applies a row-wise Jacobian penalty to the posterior latent distribution to encourage locally smooth transition learning. We show that this penalty can be interpreted as the continuous-latent analog of finite-difference smoothing of transition laws in discrete embedded-state MDPs, and estimate it efficiently using Hutchinson-style stochastic probes. Empirically, across DeepMind Control proprioceptive tasks, GPLD improves aggregate sample efficiency, with particularly strong gains on higher-complexity locomotion environments. On more challenging quadruped tasks, GPLD reaches high-return behavior earlier and exhibits more consistent late-stage learning over longer horizons. Explicit local smoothness regularization is a simple and effective way to improve latent world models for smooth continuous control environments. Code for GPLD is available at github.com/romils9/gpld-mbrl .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。