用扩散模型生成带动作的完整轨迹，提升离线强化学习效果
DAWM: Diffusion Action World Models for Offline Reinforcement Learning via Action-Inferred Transitions
- 通过条件扩散模型生成状态-奖励轨迹并反推动作
- 在D4RL上使TD3BC、IQL等算法性能超越现有扩散模型基线
- 模块化设计兼顾效率与兼容性，适合主流离线强化学习框架
基于扩散的世界模型在离线强化学习中展现出生成长时序轨迹的强大能力。然而，许多现有方法无法直接生成动作，限制了其与依赖单步时序差分（TD）学习的标准价值型离线强化学习算法的兼容性。尽管先前工作尝试联合建模状态、奖励和动作以解决此问题，但通常导致训练复杂度上升且实际性能下降。我们提出DAWM，一种基于扩散的世界模型，可基于当前状态、动作和未来回报条件生成未来状态-奖励轨迹，并结合逆动力学模型（IDM）高效推断动作。该模块化设计生成完整的合成转移数据，适用于单步TD学习，实现高效且有效的训练。实验表明，保守型离线强化学习算法如TD3BC和IQL在使用这些增强轨迹训练后表现显著提升，在D4RL基准上的多个任务中持续优于先前的扩散模型基线。
原文摘要 · Abstract (English)
Diffusion-based world models have demonstrated strong capabilities in synthesizing realistic long-horizon trajectories for offline reinforcement learning (RL). However, many existing methods do not directly generate actions alongside states and rewards, limiting their compatibility with standard value-based offline RL algorithms that rely on one-step temporal difference (TD) learning. While prior work has explored joint modeling of states, rewards, and actions to address this issue, such formulations often lead to increased training complexity and reduced performance in practice. We propose \textbf{DAWM}, a diffusion-based world model that generates future state-reward trajectories conditioned on the current state, action, and return-to-go, paired with an inverse dynamics model (IDM) for efficient action inference. This modular design produces complete synthetic transitions suitable for one-step TD-based offline RL, enabling effective and computationally efficient training. Empirically, we show that conservative offline RL algorithms such as TD3BC and IQL benefit significantly from training on these augmented trajectories, consistently outperforming prior diffusion-based baselines across multiple tasks in the D4RL benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。