arXiv:2505.10881cs.LG2025-05NeurIPS被引 13

用可学习先验替代扩散模型默认分布,高效生成高价值轨迹

Prior-Guided Diffusion Planning for Offline Reinforcement Learning

  • 用可学习分布替代标准高斯先验,通过行为正则化优化
  • 在多个长周期离线强化学习基准上超越当前最佳扩散策略
  • 无需多轮采样选择,推理成本低,适合复杂决策任务

扩散模型因能从静态数据集中有效学习高性能、泛化能力强的策略,近年来在离线强化学习中备受关注。基于扩散的规划器通过迭代去噪生成高质量轨迹,实现长周期决策。然而,现有引导采样策略如分类器引导、无分类器引导和蒙特卡洛采样选择,或产生次优多模态动作,或面临分布漂移问题,或带来高昂推理成本。为此,我们提出先验引导(Prior Guidance, PG),将行为克隆扩散模型的标准高斯先验替换为可学习分布,并通过行为正则化目标进行优化。PG直接生成高价值轨迹,无需对扩散模型本身进行昂贵的奖励优化,也无需在推理时采样多个候选进行选择。我们设计了一种高效的潜空间行为正则化训练策略,并在多个长周期离线强化学习基准上实证表明,PG显著优于当前最先进的扩散策略与规划器。代码已开源:https://github.com/ku-dmlab/PG。

原文摘要 · Abstract (English)

Diffusion models have recently gained prominence in offline reinforcement learning due to their ability to effectively learn high-performing, generalizable policies from static datasets. Diffusion-based planners facilitate long-horizon decision-making by generating high-quality trajectories through iterative denoising, guided by return-maximizing objectives. However, existing guided sampling strategies such as Classifier Guidance, Classifier-Free Guidance, and Monte Carlo Sample Selection either produce suboptimal multi-modal actions, struggle with distributional drift, or incur prohibitive inference-time costs. To address these challenges, we propose Prior Guidance (PG), a novel guided sampling framework that replaces the standard Gaussian prior of a behavior-cloned diffusion model with a learnable distribution, optimized via a behavior-regularized objective. PG directly generates high-value trajectories without costly reward optimization of the diffusion model itself, and eliminates the need to sample multiple candidates at inference for sample selection. We present an efficient training strategy that applies behavior regularization in latent space, and empirically demonstrate that PG outperforms state-of-the-art diffusion policies and planners across diverse long-horizon offline RL benchmarks.Our code is available at https://github.com/ku-dmlab/PG.

扩散模型离线RL轨迹生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。