通过显式路径修正提升流模型强化学习的稳定性与精度。
ReFPO: Reflow Regularization for Flow Matching Policy Gradients
- 在流匹配策略梯度中引入显式路径修正正则化，仅需一行代码。
- 减少代理比率峰值，稳定PPO风格训练，支持高保真单步推理。
- 适用于复杂物理仿真中的生成式策略，提升多任务鲁棒性。
我们提出重流正则化流匹配策略梯度（ReFPO），一种简单的在线强化学习方法，通过在流匹配策略梯度（FPO）中加入显式的重流正则化，实现高效的基于流的控制。我们揭示了一个关键结构特性：流匹配策略梯度中的梯度更新可被解释为隐式的优势加权重流过程，为基于流的策略梯度提供了新的几何视角。基于此洞察，ReFPO引入一个显式几何正则项，仅需一行代码即可实现，无需额外计算开销或辅助蒸馏阶段。通过结合优势引导更新与路径校正，该方法有效降低CFM代理比率的峰值，稳定PPO风格训练，并实现高保真单步推理，其性能常优于或多于多步推理。实验表明，ReFPO在GridWorld、MuJoCo Playground和高维Humanoid Control任务中均显著提升平均性能与离散化鲁棒性，为复杂物理仿真中的生成式策略提供了一种可扩展且稳定的解决方案。
原文摘要 · Abstract (English)
We present Reflow-regularized Flow Matching Policy Gradients (ReFPO), a simple online RL method that adds explicit Reflow regularization to FPO for efficient flow-based control. We uncover a key structural property: the gradient updates in Flow Matching Policy Gradients (FPO) can be interpreted as an implicit advantage-weighted Reflow process, providing a new geometric perspective on flow-based policy gradients. Building on this insight, ReFPO introduces an explicit geometric regularizer that can be implemented with a single line of code change without incurring additional computational overhead or auxiliary distillation stages. By synergizing advantage-guided updates with path rectification, our method reduces CFM proxy-ratio spikes, stabilizes PPO-style training, and enables high-fidelity one-step inference that often matches or exceeds multi-step performance. We experimentally demonstrate that ReFPO improves average performance and discretization robustness across GridWorld, MuJoCo Playground, and high-dimensional Humanoid Control tasks, providing a scalable and stable approach for generative policies in complex physical simulations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。