用增量动作替代绝对动作,让视觉强化学习更稳定高效。
ResWM: Residual-Action World Model for Visual RL
- 将控制变量改为相对前一步的增量动作,降低搜索空间
- 在深脑控制套件上提升采样效率与最终收益,控制更平滑
- 适合机器人等需稳定、节能动作的实际部署场景
从原始视觉观测中学习预测性世界模型是强化学习中的核心挑战,尤其在机器人和连续控制任务中。传统基于模型的RL框架直接以绝对动作作为未来预测条件,导致优化不稳定:最优动作分布依赖任务、事先未知,常引发振荡或低效控制。为此,本文提出残差动作世界模型(ResWM),将控制变量从绝对动作改为相对于前一步的增量调整。这一设计契合真实控制的平滑特性,缩小有效搜索空间,提升长时程规划稳定性。为增强表征能力,引入观察差异编码器,显式建模相邻帧变化,生成与残差动作天然耦合的紧凑潜在动态。ResWM以极小改动融入Dreamer风格的潜在动力学模型,无需额外超参数。想象推演与策略优化均在残差动作空间中进行,实现更平滑探索、更低控制方差与更可靠的规划。实验证明,其在DeepMind Control Suite上持续提升采样效率、最终回报与控制平滑度,显著超越Dreamer与TD-MPC等强基线。此外,该方法产生更稳定、能耗更低的动作轨迹,对实际机器人部署至关重要。结果表明,残差动作建模为算法进步与机器人实际需求之间的桥梁提供了简单而强大的范式。
原文摘要 · Abstract (English)
Learning predictive world models from raw visual observations is a central challenge in reinforcement learning (RL), especially for robotics and continuous control. Conventional model-based RL frameworks directly condition future predictions on absolute actions, which makes optimization unstable: the optimal action distributions are task-dependent, unknown a priori, and often lead to oscillatory or inefficient control. To address this, we introduce the Residual-Action World Model (ResWM), a new framework that reformulates the control variable from absolute actions to residual actions -- incremental adjustments relative to the previous step. This design aligns with the inherent smoothness of real-world control, reduces the effective search space, and stabilizes long-horizon planning. To further strengthen the representation, we propose an Observation Difference Encoder that explicitly models the changes between adjacent frames, yielding compact latent dynamics that are naturally coupled with residual actions. ResWM is integrated into a Dreamer-style latent dynamics model with minimal modifications and no extra hyperparameters. Both imagination rollouts and policy optimization are conducted in the residual-action space, enabling smoother exploration, lower control variance, and more reliable planning. Empirical results on the DeepMind Control Suite demonstrate that ResWM achieves consistent improvements in sample efficiency, asymptotic returns, and control smoothness, significantly surpassing strong baselines such as Dreamer and TD-MPC. Beyond performance, ResWM produces more stable and energy-efficient action trajectories, a property critical for robotic systems deployed in real-world environments. These findings suggest that residual action modeling provides a simple yet powerful principle for bridging algorithmic advances in RL with the practical requirements of robotics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。