提出统一生成轨迹策略,兼顾速度与性能,突破离线强化学习瓶颈
Offline Reinforcement Learning with Generative Trajectory Policies
- 将扩散、流匹配等生成模型统一为求解ODE的连续轨迹方法
- 在D4RL上超越已有生成策略,多个难任务达满分表现
- 适合关注高效高精度离线强化学习的研究者与工程实践者
生成模型因其捕捉复杂多模态行为的能力,已成为离线强化学习中的强大策略工具。然而,现有方法面临显著权衡:如扩散策略等迭代模型计算成本高,而一致性策略等单步模型性能常下降。本文指出,通过将现代生成模型(包括扩散、流匹配和一致性模型)统一视为由常微分方程(ODE)控制的连续时间生成轨迹,可弥合这一差距。该原理性框架为生成策略设计提供了更清晰的思路,并促使我们提出生成轨迹策略(GTP),一种学习底层ODE完整解映射的新范式。为使该范式适用于离线强化学习,我们进一步引入两种理论上有保障的改进。实验表明,GTP在D4RL基准上达到顶尖性能,显著优于以往生成策略,在多个极具挑战性的AntMaze任务中实现满分。
原文摘要 · Abstract (English)
Generative models have emerged as a powerful class of policies for offline reinforcement learning (RL) due to their ability to capture complex, multi-modal behaviors. However, existing methods face a stark trade-off: slow, iterative models like diffusion policies are computationally expensive, while fast, single-step models like consistency policies often suffer from degraded performance. In this paper, we demonstrate that it is possible to bridge this gap. The key to moving beyond the limitations of individual methods, we argue, lies in a unifying perspective that views modern generative models, including diffusion, flow matching, and consistency models, as specific instances of learning a continuous-time generative trajectory governed by an Ordinary Differential Equation (ODE). This principled foundation provides a clearer design space for generative policies in RL and allows us to propose Generative Trajectory Policies (GTPs), a new and more general policy paradigm that learns the entire solution map of the underlying ODE. To make this paradigm practical for offline RL, we further introduce two key theoretically principled adaptations. Empirical results demonstrate that GTP achieves state-of-the-art performance on D4RL benchmarks - it significantly outperforms prior generative policies, achieving perfect scores on several notoriously hard AntMaze tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。