新强化学习算法让扩散策略在线训练更稳定高效。
D2 Actor Critic: Diffusion Actor Meets Distributional Critic
- 用分布式批评机制替代传统梯度，降低训练波动。
- 在18个复杂任务上达到顶尖性能，涵盖人形、狗、机械手等场景。
- 适合需要高鲁棒性和泛化能力的机器人控制任务。
我们提出D2AC，一种新型无模型强化学习算法，旨在高效在线训练表达性强的扩散策略。核心是避免传统策略梯度的高方差和时间反向传播的复杂性，通过稳定的学习过程实现有效优化。这一过程的关键在于第二个贡献：融合分布式强化学习与裁剪双Q学习设计的鲁棒分布式批评者。该算法在包含18个高难度任务的基准测试中表现卓越，涵盖密集奖励和目标条件化任务，如人形（Humanoid）、狗（Dog）和影子手（Shadow Hand）等环境。此外，还评估了一个生物启发的捕食-猎物任务，以检验方法的鲁棒性和泛化能力。代码已公开。
原文摘要 · Abstract (English)
We introduce D2AC, a new model-free reinforcement learning (RL) algorithm designed to train expressive diffusion policies online effectively. At its core is a policy improvement objective that avoids the high variance of typical policy gradients and the complexity of backpropagation through time. This stable learning process is critically enabled by our second contribution: a robust distributional critic, which we design through a fusion of distributional RL and clipped double Q-learning. The resulting algorithm is highly effective, achieving state-of-the-art performance on a benchmark of eighteen hard RL tasks, including Humanoid, Dog, and Shadow Hand domains, spanning both dense-reward and goal-conditioned RL scenarios. Beyond standard benchmarks, we also evaluate a biologically motivated predator-prey task to examine the behavioral robustness and generalization capacity of our approach. Code: https://github.com/d2ac-actor-critic/d2ac-public
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。