让流模型机器人学会自主任务，突破强化学习训练难题
$π_\texttt{RL}$: Online RL Fine-tuning for Flow-based Vision-Language-Action Models
- 用可学习噪声网络精确计算动作对数似然，解决流模型不可导问题
- 通过两层MDP与随机微分方程转换，实现高效探索与策略优化
- 在多种分布下显著提升性能，适合需自主决策的机器人系统
视觉-语言-动作（VLA）模型使机器人能够从多模态输入中理解并执行复杂任务。尽管已有研究尝试使用强化学习（RL）自动化监督微调（SFT）中繁琐的数据收集过程，但将RL应用于大规模基于流的VLA（如π₀、π₀.₅）仍面临挑战，主要源于流匹配带来的难以处理的动作对数似然。为此，我们提出πₜᵣₗ，采用两项关键技术：（1）流-噪声（Flow-Noise）将去噪过程建模为带可学习噪声网络的离散时间马尔可夫决策过程，实现精确的对数似然计算；（2）流-SDE将去噪与智能体-环境交互结合，构建双层马尔可夫决策过程，并通过ODE到SDE转换实现高效的强化学习探索。我们在多个基准上评估πₜᵣₗ，实验表明，在分布内和分布外设置下，强化学习均带来显著性能提升。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models enable robots to understand and perform complex tasks from multimodal input. Although recent work explores using reinforcement learning (RL) to automate the laborious data collection process in scaling supervised fine-tuning (SFT), applying RL to large-scale flow-based VLAs (\eg, $π_0$, $π_{0.5}$) remains challenging due to intractable action log-likelihoods raised from flow matching. We address this challenge with $π_{\texttt{RL}}$, featuring two technical approaches: (1) \textbf{Flow-Noise} models the denoising process as a discrete-time MDP with a learnable noise network for exact log-likelihood computation. (2) \textbf{Flow-SDE} integrates denoising with agent-environment interaction, formulating a two-layer MDP that employs ODE-to-SDE conversion for efficient RL exploration. We evaluate $π_{\texttt{RL}}$ across various benchmarks, with experiments demonstrating that RL yields significant performance improvements in both in-distribution and out-of-distribution settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。