提出1步采样方法，让扩散策略推理速度提升数个数量级
One-Step Flow Policy Mirror Descent
- 基于流匹配理论，实现单步采样而无需额外训练
- 在MuJoCo和DeepMind Control上性能接近基线，计算成本降低数十倍
- 适合对响应速度要求高的在线强化学习场景
扩散策略因强大的表达能力在在线强化学习中表现优异，但其推理依赖缓慢的迭代采样过程，限制了响应速度。为克服此问题，我们提出流策略镜面下降（FPMD），一种支持流策略推理时1步采样的在线强化学习算法。该方法利用直插式流匹配模型中分布方差与单步采样离散化误差之间的理论联系，无需额外蒸馏或一致性训练。我们基于修正流策略和均值流策略分别提出了两种算法变体。在MuJoCo和视觉版DeepMind Control Suite上的大量实验表明，所提算法在性能上可媲美扩散策略基线，同时推理阶段计算成本降低数个数量级。
原文摘要 · Abstract (English)
Diffusion policies have achieved great success in online reinforcement learning (RL) due to their strong expressive capacity. However, the inference of diffusion policy models relies on a slow iterative sampling process, which limits their responsiveness. To overcome this limitation, we propose Flow Policy Mirror Descent (FPMD), an online RL algorithm that enables 1-step sampling during flow policy inference. Our approach exploits a theoretical connection between the distribution variance and the discretization error of single-step sampling in straight interpolation flow matching models, and requires no extra distillation or consistency training. We present two algorithm variants based on rectified flow policy and MeanFlow policy, respectively. Extensive empirical evaluations on MuJoCo and visual DeepMind Control Suite benchmarks demonstrate that our algorithms show strong performance comparable to diffusion policy baselines while requiring orders of magnitude less computational cost during inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。