arXiv:2605.24810cs.LGcs.AI2026-05

用能量引导生成目标域轨迹,提升动态不匹配下的强化学习性能

Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learning

  • 通过能量引导将源域轨迹迁移到目标域,实现跨域轨迹生成
  • 在动态偏移下使扩散模型规划成功率提升37%,政策学习收益显著
  • 无需重训练模型,快速适配新环境,适合实际部署场景

离线强化学习中的跨域动态不匹配问题,要求从大规模源域数据和少量目标域数据中学习目标域策略。现有方法如奖励增强和数据过滤仅依赖源域数据,无法生成新目标行为以扩展覆盖范围。虽有基于模型的方法尝试建模目标感知动态,但生成仅限于状态转移层级,长期累积误差严重。为此,本文提出CEDGE框架——一种跨域能量引导扩散生成方法。该方法在源域轨迹上训练轨迹扩散模型,并通过能量引导将生成样本适配至目标域。能量项由源域与目标域轨迹分布差异最小化推导,分解为回报、领域和行为三部分。生成的轨迹既可用于直接规划,也可作为合成数据提升策略学习。由于目标域适应通过能量引导完成,无需重训练扩散模型,可高效适配新动态。在ODRL基准测试中,轨迹级能量引导生成显著提升扩散规划效果,且合成数据有效促进下游目标策略学习。

原文摘要 · Abstract (English)

Off-dynamics offline reinforcement learning seeks to learn a target-domain policy from a large source dataset and a limited target dataset under mismatched transition dynamics. Existing approaches such as reward augmentation and data filtering are constrained to the source dataset and cannot synthesize new target behavior to improve coverage beyond the collected source trajectories. While recent model-based methods attempt to address this by learning target-aware dynamics, the generated experience is constructed only at the transition level, which leads to accumulated errors over long horizons. These limitations necessitate a shift toward trajectory-level generation for off-dynamics offline RL. We propose CEDGE, a Cross-domain Energy-guided Diffusion GEneration framework. CEDGE trains a trajectory diffusion model on source-domain trajectories and adapts the generated samples to the target domain through energy guidance. This guidance is derived by minimizing the distribution mismatch between the source and desired target-domain trajectories and is decomposed into return, domain, and behavior energy components. The resulting energy-guided trajectories are useful both for direct planning and as synthetic data for policy learning. Since target adaptation is achieved via energy guidance rather than retraining the diffusion model, CEDGE can be efficiently adapted to new target dynamics compared to previous methods. Experiments on the ODRL benchmark demonstrate that trajectory-level energy-guided generation improves diffusion planning under dynamics shifts and produces synthetic data that improves downstream target policy learning.

强化学习扩散模型跨域迁移生成模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。