用单步生成实现高效强化学习，速度远超传统扩散模型。
Score-Based One-step MeanFlow Policy Optimization
- 通过评分估计与概率流微分方程，从价值函数构建目标速度场。
- 在线强化学习中仅用一步生成即达前沿性能，训练推理更快。
- 适合追求高速生成的在线决策场景，如机器人控制。
扩散模型和流匹配已成为强化学习中的表达性强策略类，但其依赖多步去噪导致推理时开销大，尤其在在线强化学习中问题突出。平均速度场（MeanFlow）提供了一种替代方案，通过单一网络评估将噪声映射到数据。然而，传统 MeanFlow 需要目标分布的样本构建目标速度场，在线强化学习中不可得。我们提出基于评分的单步平均速度场策略优化（SOM），通过评分估计与概率流常微分方程，直接从 Q 函数构造目标速度场，从而将概率质量集中于高价值模式。在完全在线强化学习设置下，SOM 仅需单步生成即可在运动任务上达到当前最优表现，且显著降低训练与推理时间，相比先前基于扩散与流匹配的策略有明显优势。
原文摘要 · Abstract (English)
Diffusion and flow matching have emerged as expressive policy classes in reinforcement learning, but their reliance on multi-step denoising imposes substantial computational overhead at inference time, which is particularly problematic in online RL. MeanFlow offers a promising alternative by learning an average velocity field that maps noise to data in a single network evaluation. However, MeanFlow typically requires samples from the target distribution to construct its target velocity field, which are unavailable in online RL. We propose Score-Based One-step MeanFlow Policy Optimization (SOM), an actor-critic algorithm that resolves this by constructing the target velocity field directly from the Q-function via score estimation and a probability flow ODE, thereby concentrating probability mass on high-value modes. In the fully online RL setting, SOM achieves state-of-the-art performance on locomotion tasks with a single generation step, while substantially reducing both training and inference time compared to prior diffusion- and flow-matching-based policies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。