提出首个异步强化学习框架，提升视觉-语言-动作模型训练效率。
RL-VLA$^3$: A Flexible and Asynchronous Reinforcement Learning Framework for VLA Training
- 采用动态批处理与灵活环境分片，实现仿真、推理、训练的细粒度异步交互。
- 在8到256张GPU上测试，吞吐量比同步基线最高提升85.2%。
- 适合需要高效训练视觉-语言-动作模型的研究者与工业开发者。
强化学习已成为后训练视觉-语言-动作（VLA）模型的关键范式，使具身智能体通过与环境互动实现适应与优化。然而，现有VLA强化学习框架沿用传统大模型训练的同步设计，将完整轨迹视为不可分割单元，并严格交替数据收集与策略优化，这与VLA训练的独特性严重不符，因物理模拟器引入高度可变且资源密集的延迟。为此，我们提出RL-VLA$^3$，一个全异步分布式强化学习框架，通过动态批处理调度器与灵活环境分片策略，实现仿真、推理与训练组件间的细粒度异步交互。在多种模拟后端、VLA架构及强化学习算法上的广泛实验表明，RL-VLA$^3$在保持相同样本效率的前提下，吞吐量相较同步基线最高提升85.2%，并验证了从8到256张GPU的可扩展性。据我们所知，这是首个专为VLA训练系统级挑战量身定制的全异步强化学习框架。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has emerged as a critical paradigm for post-training Vision-Language-Action (VLA) models, enabling embodied agents to adapt and improve through environmental interaction. However, existing RL frameworks for VLAs inherit synchronous design principles from traditional LLM training, treating entire rollouts as indivisible units and alternating strictly between data collection and policy optimization. This fundamentally mismatches the unique characteristics of VLA training, as physical simulators introduce highly variable, resource-intensive latencies. To address this, we introduce RL-VLA$^3$, a fully asynchronous distributed RL framework that enables fine-grained asynchronous interaction between simulation, inference, and training components through dynamic batching schedulers and flexible environment sharding strategies. Extensive experiments across diverse simulation backends, VLA architectures, and RL algorithms demonstrate that RL-VLA$^3$ achieves throughput improvements of up to 85.2\% over synchronous baselines while maintaining identical sample efficiency, with scalability validated from 8 to 256 GPUs. To our knowledge, RL-VLA$^3$ is the first fully asynchronous RL training framework tailored specifically for the system-level challenges of VLA training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。