改进强化学习在随机环境中的梯度学习，提升样本效率。
Distributional value gradients for stochastic environments
- 用分布式方法同时建模价值函数及其梯度，引入Sobolev训练框架。
- 在MuJoCo任务中表现优于传统方法，尤其在噪声环境下更稳定。
- 适合需要高鲁棒性的连续控制场景，如机器人控制与仿真应用。
基于梯度正则化的价值学习方法通过利用对转移动态和奖励的模型来估计回报梯度，从而提升样本效率。然而，现有方法（如MAGE）在随机或噪声环境中表现不佳，限制了其应用范围。本文通过将分布式强化学习扩展至连续状态-动作空间，不仅建模标量状态-动作价值函数的分布，还建模其梯度分布，提出分布式Sobolev训练（Distributional Sobolev Training）。受随机价值梯度（SVG）启发，该方法采用条件变分自编码器（cVAE）实现一步世界模型，对奖励和转移分布进行建模。所提框架为基于样本的方法，使用最大切片最大均值差异（MSMMD）构造分布式贝尔曼算子。我们证明了增强梯度的贝尔曼算子是收缩映射，具有唯一不动点，并揭示了梯度感知强化学习中收缩性与平滑性的根本权衡。通过在简单随机强化学习玩具问题上的验证，以及在多个MuJoCo环境中的基准测试，展示了本方法的有效性。
原文摘要 · Abstract (English)
Gradient-regularized value learning methods improve sample efficiency by leveraging learned models of transition dynamics and rewards to estimate return gradients. However, existing approaches, such as MAGE, struggle in stochastic or noisy environments, limiting their applicability. In this work, we address these limitations by extending distributional reinforcement learning on continuous state-action spaces to model not only the distribution over scalar state-action value functions but also over their gradients. We refer to this approach as Distributional Sobolev Training. Inspired by Stochastic Value Gradients (SVG), our method utilizes a one-step world model of reward and transition distributions implemented via a conditional Variational Autoencoder (cVAE). The proposed framework is sample-based and employs Max-sliced Maximum Mean Discrepancy (MSMMD) to instantiate the distributional Bellman operator. We prove that the Sobolev-augmented Bellman operator is a contraction with a unique fixed point, and highlight a fundamental smoothness trade-off underlying contraction in gradient-aware RL. To validate our method, we first showcase its effectiveness on a simple stochastic reinforcement learning toy problem, then benchmark its performance on several MuJoCo environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。