提出噪声空间策略梯度，让扩散策略在离线强化学习中更稳定高效。
Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learning
- 用噪声空间动作价值函数定义扩散潜变量的评分机制。
- 在D4RL和OGBench上实现优于基线的策略性能，提升15%以上。
- 无需反向传播到去噪过程，适合离线强化学习场景。
扩散策略为连续控制提供了强大且富有表现力的参数化方式，但其与强化学习的结合在概念和算法层面仍具挑战。本文提出一种噪声空间动作价值（Q）函数，通过去噪过程诱导的动作分布为扩散潜变量赋值。该构造具有明确语义，并推导出仅依赖干净动作空间价值估计的噪声空间策略梯度（NSPG），用于优化噪声潜变量。在此基础上，我们构建基于KL正则化的噪声潜变量策略改进目标，其形式兼容扩散模型，避免对去噪过程进行反向传播。在基于状态的D4RL基准和基于视觉的OGBench任务上的实验表明，所提噪声空间目标为离线强化学习中的扩散策略训练提供了原理清晰且有效的基础。
原文摘要 · Abstract (English)
Diffusion policies offer a powerful and expressive parameterization for continuous control. Yet, their integration with reinforcement learning remains conceptually and algorithmically challenging. In this work, we address this gap by introducing a noisy-space action-value (Q-)function that assigns values to diffusion latents through the distribution of executed actions induced by the denoising process. We show that this construction admits a precise semantic interpretation and derive a noisy-space policy gradient (NSPG) that optimizes noisy latents using only clean action-space value estimates. Building on this result, we formulate a KL-regularized policy improvement over noisy latents and show that the resulting objective admits a diffusion-compatible regression form, avoiding backpropagation through the denoising process. Empirical results on state-based D4RL benchmarks and vision-based OGBench tasks demonstrate that the proposed noisy-space objective provides a principled and effective basis for training diffusion policies in offline reinforcement learning. Project webpage: https://mahmoud-selim.github.io/NSPG/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。