用强化学习提升视觉语言动作模型的长程规划能力，减少对海量人工数据依赖。
SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning
- 基于强化学习设计专用轨迹采样与并行化策略，高效训练视觉语言动作模型。
- 在LIBERO和RoboTwin上超越监督微调，实测表现达当前最佳水平。
- 发现新现象'pushcut'，政策可学会训练中未见的新动作模式，适合机器人规划研究者。
视觉-语言-动作（VLA）模型已成为机器人操作的有力范式。尽管大规模预训练和监督微调（SFT）取得显著进展，仍面临两大挑战：(i) SFT扩展所需的大规模人工操作轨迹稀缺且成本高昂；(ii) 对分布外任务泛化能力有限。近期大型推理模型（LRM）突破表明，强化学习（RL）可显著增强逐步推理能力，这引出一个问题：能否用RL同样提升VLA的长程动作规划？本文提出SimpleVLA-RL，一种专为VLA设计的高效强化学习框架。在veRL基础上，引入VLA特有轨迹采样、可扩展并行化、多环境渲染及优化损失计算。应用于OpenVLA-OFT时，SimpleVLA-RL在LIBERO上达到当前最优性能，并在RoboTwin 1.0&2.0上超越$π_0$，得益于引入的探索增强策略。该方法不仅降低对大规模数据依赖，实现强泛化能力，还在真实任务中显著优于SFT。此外，我们观察到一种新现象‘pushcut’，即策略在训练中发现了此前未见的动作模式。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models have recently emerged as a powerful paradigm for robotic manipulation. Despite substantial progress enabled by large-scale pretraining and supervised fine-tuning (SFT), these models face two fundamental challenges: (i) the scarcity and high cost of large-scale human-operated robotic trajectories required for SFT scaling, and (ii) limited generalization to tasks involving distribution shift. Recent breakthroughs in Large Reasoning Models (LRMs) demonstrate that reinforcement learning (RL) can dramatically enhance step-by-step reasoning capabilities, raising a natural question: Can RL similarly improve the long-horizon step-by-step action planning of VLA? In this work, we introduce SimpleVLA-RL, an efficient RL framework tailored for VLA models. Building upon veRL, we introduce VLA-specific trajectory sampling, scalable parallelization, multi-environment rendering, and optimized loss computation. When applied to OpenVLA-OFT, SimpleVLA-RL achieves SoTA performance on LIBERO and even outperforms $π_0$ on RoboTwin 1.0\&2.0 with the exploration-enhancing strategies we introduce. SimpleVLA-RL not only reduces dependence on large-scale data and enables robust generalization, but also remarkably surpasses SFT in real-world tasks. Moreover, we identify a novel phenomenon ``pushcut'' during RL training, wherein the policy discovers previously unseen patterns beyond those seen in the previous training process. Github: https://github.com/PRIME-RL/SimpleVLA-RL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。