用扩散模型指导强化学习，让智能体更聪明地做长期决策。
DIAR: Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation
- 用扩散模型生成动作序列，结合价值函数动态调整决策长度。
- 在Maze2D等任务中，性能超越现有最优方法，尤其在稀疏奖励场景下。
- 适合需要长程规划的复杂任务，如机器人控制与导航。
我们提出一种新型离线强化学习框架DIAR（Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation），解决离线RL中的分布外样本和长时序问题。通过扩散模型学习状态-动作序列分布，并引入值函数实现更均衡、自适应的决策。DIAR设计了自适应重评估机制，通过比较当前与未来状态值动态调整决策长度，支持灵活的长期决策。同时，结合扩散模型引导的价值函数，缓解Q值过高估计问题。扩散模型生成多样化潜在轨迹，提升策略鲁棒性与泛化能力。在Maze2D、AntMaze和Kitchen等任务中，DIAR持续优于当前最优算法，尤其在长时序、稀疏奖励环境中表现突出。
原文摘要 · Abstract (English)
We propose a novel offline reinforcement learning (offline RL) approach, introducing the Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation (DIAR) framework. We address two key challenges in offline RL: out-of-distribution samples and long-horizon problems. We leverage diffusion models to learn state-action sequence distributions and incorporate value functions for more balanced and adaptive decision-making. DIAR introduces an Adaptive Revaluation mechanism that dynamically adjusts decision lengths by comparing current and future state values, enabling flexible long-term decision-making. Furthermore, we address Q-value overestimation by combining Q-network learning with a value function guided by a diffusion model. The diffusion model generates diverse latent trajectories, enhancing policy robustness and generalization. As demonstrated in tasks like Maze2D, AntMaze, and Kitchen, DIAR consistently outperforms state-of-the-art algorithms in long-horizon, sparse-reward environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。