异步框架提升视觉语言动作模型训练效率，大幅降低数据成本。
AcceRL: A Distributed Asynchronous Reinforcement Learning and World Model Framework for Vision-Language-Action Models
- 分离环境运行、推理与梯度更新，消除同步延迟
- 吞吐量比主流同步方法快2.4倍，样本效率提升200倍
- 支持插件式世界模型，适合大规模具身智能研究
大规模视觉-语言-动作（VLA）模型的强化学习受限于同步瓶颈和环境数据获取的高成本。为此，我们提出AcceRL，一个分布式异步强化学习框架，通过物理隔离环境采样、模型推理和梯度更新，消除同步系统固有的长尾空闲延迟，最大化硬件利用率并实现可扩展吞吐。该框架模块化设计，支持多种即插即用的世界模型集成。实验表明，基础框架在全部四个LIBERO任务套件上表现优异；系统层面，异步架构相较领先同步基线实现2.4倍吞吐加速；算法层面，利用预训练于1,000条离线轨迹的世界模型，AcceRL在LIBERO-Spatial上实现高达200倍的在线样本效率提升，构建了一个兼具高样本效率与高时间效率的具身智能训练框架。代码已公开于GitHub。
原文摘要 · Abstract (English)
Reinforcement learning (RL) for large-scale Vision-Language-Action (VLA) models is severely bottlenecked by synchronization barriers and the high cost of environment data acquisition. To overcome these challenges, we propose AcceRL, a distributed asynchronous RL framework that physically isolates environment rollouts, model inference, and gradient updates. By eliminating the cascading long-tail idle bubbles inherent in synchronous systems, AcceRL maximizes hardware utilization and ensures scalable throughput. Furthermore, AcceRL features a modular design that supports the integration of diverse, plug-and-play world models into its distributed pipeline. Extensive experiments demonstrate that the base framework achieves highly competitive performance across all four LIBERO~\cite{liu2023libero} task suites. Systematically, the asynchronous architecture delivers a $2.4\times$ throughput speedup over leading synchronous baselines. Algorithmically, by leveraging a world model pre-trained on 1,000 offline trajectories, AcceRL achieves up to a $200\times$ improvement in online sample efficiency on LIBERO-Spatial, establishing a robust framework that is both sample-efficient and time-efficient for embodied AI. Code is included in the supplementary material. Code is available at https://github.com/distanceLu/AcceRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。