用在线强化学习微调流模型，少步数也能高效控制机器人
ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning
- 将流模型转为马尔可夫过程，通过可学习噪声实现稳定探索
- 在少至1步去噪下，奖励提升超135%，比先进方法快82.63%
- 适合需要快速推理的机器人控制场景，尤其视觉与稀疏奖励任务
我们提出ReinFlow，一种简单而有效的在线强化学习框架，用于微调连续机器人控制中的流匹配策略。基于严格的强化学习理论，ReinFlow 在流策略的确定路径中注入可学习噪声，将流转换为离散时间马尔可夫过程，实现精确且直接的似然计算。这一转换促进探索并确保训练稳定性，使ReinFlow 能微调多种流模型变体，包括Rectified Flow [35] 和 Shortcut Models [19]，即使在极少数或仅一步去噪的情况下也有效。我们在代表性步态行走和操作任务中进行基准测试，包括带视觉输入的长时程规划与稀疏奖励环境。经ReinFlow微调后，Rectified Flow策略在复杂腿式行走任务中平均奖励提升135.36%，去噪步数减少，耗时节省82.63%（相较最先进扩散强化学习方法DPPO [43]）。Shortcut Model策略在状态与视觉操作任务中的成功率平均提升40.34%，在四步甚至一步去噪下表现媲美微调后的DDIM策略，平均节省23.20%计算时间。项目主页：https://reinflow.github.io/
原文摘要 · Abstract (English)
We propose ReinFlow, a simple yet effective online reinforcement learning (RL) framework that fine-tunes a family of flow matching policies for continuous robotic control. Derived from rigorous RL theory, ReinFlow injects learnable noise into a flow policy's deterministic path, converting the flow into a discrete-time Markov Process for exact and straightforward likelihood computation. This conversion facilitates exploration and ensures training stability, enabling ReinFlow to fine-tune diverse flow model variants, including Rectified Flow [35] and Shortcut Models [19], particularly at very few or even one denoising step. We benchmark ReinFlow in representative locomotion and manipulation tasks, including long-horizon planning with visual input and sparse reward. The episode reward of Rectified Flow policies obtained an average net growth of 135.36% after fine-tuning in challenging legged locomotion tasks while saving denoising steps and 82.63% of wall time compared to state-of-the-art diffusion RL fine-tuning method DPPO [43]. The success rate of the Shortcut Model policies in state and visual manipulation tasks achieved an average net increase of 40.34% after fine-tuning with ReinFlow at four or even one denoising step, whose performance is comparable to fine-tuned DDIM policies while saving computation time for an average of 23.20%. Project webpage: https://reinflow.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。