用少步流模型替代扩散模型，提升强化学习训练与推理效率。
Mean Flow Policy Optimization
- 采用少步流模型表示策略，减少生成步骤
- 在多个基准上达到或超过扩散模型性能，训练推理更快
- 适合追求高效强化学习的科研与工程应用
扩散模型最近作为在线强化学习中的表达性策略表示兴起，但其迭代生成过程带来了显著的训练和推理开销。为克服这一限制，我们提出使用均值流（MeanFlow）模型——一类少步流基生成模型——来表示策略，以提升相较于扩散型强化学习方法的训练与推理效率。为促进探索，我们在最大熵强化学习框架下通过软策略迭代优化均值流策略，并解决均值流策略特有的两个关键挑战：动作似然评估与软策略改进。在MuJoCo、DeepMind Control Suite和HumanoidBench基准上的实验表明，我们的方法——均值流策略优化（MFPO）——在性能上可媲美或超越当前扩散基基线，同时大幅降低训练与推理时间。代码已公开于https://github.com/dongxiaoyi-xyz/MFPO。
原文摘要 · Abstract (English)
Diffusion models have recently emerged as expressive policy representations for online reinforcement learning (RL). However, their iterative generative processes introduce substantial training and inference overhead. To overcome this limitation, we propose to represent policies using MeanFlow models, a class of few-step flow-based generative models, to improve training and inference efficiency over diffusion-based RL approaches. To promote exploration, we optimize MeanFlow policies under the maximum entropy RL framework via soft policy iteration, and address two key challenges specific to MeanFlow policies: action likelihood evaluation and soft policy improvement. Experiments on MuJoCo, DeepMind Control Suite and HumanoidBench benchmarks demonstrate that our method, Mean Flow Policy Optimization (MFPO), achieves performance comparable to or exceeding current diffusion-based baselines while considerably reducing training and inference time. Our code is available at https://github.com/dongxiaoyi-xyz/MFPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。