提出一种快速稳定的扩散规划方法，显著减少训练步数。
Fast and Stable Diffusion Planning through Variational Adaptive Weighting
- 基于变分推断设计自适应权重函数，无需神经网络近似
- 在Maze2D和Kitchen任务上仅需1/10训练步数即达竞争性能
- 适合追求高效训练的离线强化学习研究者
扩散模型在离线强化学习中展现出潜力，但常面临高训练成本与慢收敛问题，尤其使用基于Transformer的去噪主干时。尽管已有优化策略如改进噪声调度、辅助预测目标和自适应损失加权，仍难以实现稳定高效训练。现有损失加权函数多依赖神经网络近似器，在早期训练阶段因反馈稀疏导致泛化能力不足而失效。本文推导出一种变分最优的不确定性感知加权函数，并提出一种基于流模型框架的闭式多项式近似方法，实现在线估计。将该方法集成至扩散规划流程，在标准离线强化学习基准上评估。在Maze2D和Kitchen任务上的实验表明，本方法可在最多减少10倍训练步数下达到竞争力性能，验证了其实际有效性。
原文摘要 · Abstract (English)
Diffusion models have recently shown promise in offline RL. However, these methods often suffer from high training costs and slow convergence, particularly when using transformer-based denoising backbones. While several optimization strategies have been proposed -- such as modified noise schedules, auxiliary prediction targets, and adaptive loss weighting -- challenges remain in achieving stable and efficient training. In particular, existing loss weighting functions typically rely on neural network approximators, which can be ineffective in early training phases due to limited generalization capacity of MLPs when exposed to sparse feedback in the early training stages. In this work, we derive a variationally optimal uncertainty-aware weighting function and introduce a closed-form polynomial approximation method for its online estimation under the flow-based generative modeling framework. We integrate our method into a diffusion planning pipeline and evaluate it on standard offline RL benchmarks. Experimental results on Maze2D and Kitchen tasks show that our method achieves competitive performance with up to 10 times fewer training steps, highlighting its practical effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。