将流模型改造成一步生成策略,实现快速高效的动作采样。
One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlow
- 用残差结构统一速度场与噪声转动作,一步生成动作
- 在73个任务上表现优异,支持多模态动作分布建模
- 单阶段训练兼容Q-learning,适合离线强化学习场景
我们提出一种基于均值流(MeanFlow)的单步生成策略,通过残差重构使噪声直接映射为动作,兼容Q-learning。传统高斯策略虽推理快但难以捕捉复杂多模态动作分布;现有基于流的方法虽表达能力强,却通常需蒸馏与两阶段训练。为此,我们重构均值流,将速度场与噪声到动作的转换融合进单一策略网络,无需单独速度估计。我们探索多种重构变体,发现一种有效的残差形式可实现表达性强且稳定的策略学习。该方法具有三大优势:1)单步噪声到动作生成,效率高;2)能有效建模多模态动作分布;3)在单阶段训练下通过Q-learning实现稳定学习。在OGBench和D4RL基准的73个任务上广泛实验表明,该方法在离线及离线转在线强化学习中均表现强劲。代码已开源:https://github.com/HiccupRL/MeanFlowQL。
原文摘要 · Abstract (English)
We introduce a one-step generative policy for offline reinforcement learning that maps noise directly to actions via a residual reformulation of MeanFlow, making it compatible with Q-learning. While one-step Gaussian policies enable fast inference, they struggle to capture complex, multimodal action distributions. Existing flow-based methods improve expressivity but typically rely on distillation and two-stage training when trained with Q-learning. To overcome these limitations, we propose to reformulate MeanFlow to enable direct noise-to-action generation by integrating the velocity field and noise-to-action transformation into a single policy network-eliminating the need for separate velocity estimation. We explore several reformulation variants and identify an effective residual formulation that supports expressive and stable policy learning. Our method offers three key advantages: 1) efficient one-step noise-to-action generation, 2) expressive modelling of multimodal action distributions, and 3) efficient and stable policy learning via Q-learning in a single-stage training setup. Extensive experiments on 73 tasks across the OGBench and D4RL benchmarks demonstrate that our method achieves strong performance in both offline and offline-to-online reinforcement learning settings. Code is available at https://github.com/HiccupRL/MeanFlowQL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。