用流匹配提升强化学习策略，无需精确似然计算。
Flow Matching Policy Gradients
- 将流匹配损失融入策略梯度框架，实现端到端优化。
- 在连续控制任务中从零训练出性能更高的扩散策略。
- 可捕捉多模态动作分布，适合条件不足场景。
基于流的生成模型（包括扩散模型）在高维空间中建模连续分布方面表现优异。本文提出流策略优化（FPO），一种简洁的在线策略强化学习算法，将流匹配引入策略梯度框架。FPO将策略优化转化为基于条件流匹配损失的优势加权比，兼容主流PPO-clip框架。该方法避免了精确似然计算的需求，同时保留了流模型的生成能力。与以往依赖特定采样方式的扩散强化学习方法不同，FPO在训练和推理阶段对扩散或流积分方式均无偏好。实验表明，FPO可在多种连续控制任务中从零训练出扩散型策略，并发现流模型能有效捕捉多模态动作分布，在欠条件设置下表现优于高斯策略。
原文摘要 · Abstract (English)
Flow-based generative models, including diffusion models, excel at modeling continuous distributions in high-dimensional spaces. In this work, we introduce Flow Policy Optimization (FPO), a simple on-policy reinforcement learning algorithm that brings flow matching into the policy gradient framework. FPO casts policy optimization as maximizing an advantage-weighted ratio computed from the conditional flow matching loss, in a manner compatible with the popular PPO-clip framework. It sidesteps the need for exact likelihood computation while preserving the generative capabilities of flow-based models. Unlike prior approaches for diffusion-based reinforcement learning that bind training to a specific sampling method, FPO is agnostic to the choice of diffusion or flow integration at both training and inference time. We show that FPO can train diffusion-style policies from scratch in a variety of continuous control tasks. We find that flow-based models can capture multimodal action distributions and achieve higher performance than Gaussian policies, particularly in under-conditioned settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。