提出时间扩散规划器,让智能体更高效地做决策。
Efficient Diffusion Planning with Temporal Diffusion
- 将去噪步骤分布到时间维度,逐步模糊初始计划
- 每步仅用少量去噪更新计划,决策频率提升11-24.8倍
- 自动重规划机制防止计划偏离现实,适合离线强化学习
扩散规划是一种从离线数据中学习高性能策略的有前景方法。为避免规划与现实之间的差异影响性能,先前工作在每个时间步都生成新计划,但导致显著计算开销和较低决策频率,频繁切换计划也可能损害性能。受人类行为启发——短期计划详细、长期计划较泛化并随时间调整,我们提出时间扩散规划器(TDP),通过将去噪步骤分布到时间维度提升决策效率。TDP首先生成一个随时间逐渐模糊的初始计划,后续每一步仅用少量去噪步骤更新前一计划,而非生成全新计划,从而降低平均去噪步数。此外,引入自动重规划机制以防止计划与现实产生显著偏差。在D4RL上的实验表明,相比每步生成新计划的方法,TDP使决策频率提升11至24.8倍,同时达到更高或相当的性能。
原文摘要 · Abstract (English)
Diffusion planning is a promising method for learning high-performance policies from offline data. To avoid the impact of discrepancies between planning and reality on performance, previous works generate new plans at each time step. However, this incurs significant computational overhead and leads to lower decision frequencies, and frequent plan switching may also affect performance. In contrast, humans might create detailed short-term plans and more general, sometimes vague, long-term plans, and adjust them over time. Inspired by this, we propose the Temporal Diffusion Planner (TDP) which improves decision efficiency by distributing the denoising steps across the time dimension. TDP begins by generating an initial plan that becomes progressively more vague over time. At each subsequent time step, rather than generating an entirely new plan, TDP updates the previous one with a small number of denoising steps. This reduces the average number of denoising steps, improving decision efficiency. Additionally, we introduce an automated replanning mechanism to prevent significant deviations between the plan and reality. Experiments on D4RL show that, compared to previous works that generate new plans every time step, TDP improves the decision-making frequency by 11-24.8 times while achieving higher or comparable performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。