arXiv:2605.13276cs.AIcs.RO2026-05被引 2

解决视觉语言动作模型训练中的并发与延迟瓶颈,提升大规模强化学习效率。

D-VLA: A High-Concurrency Distributed Asynchronous Reinforcement Learning Framework for Vision-Language-Action Models

论文配图:D-VLA: A High-Concurrency Distributed Asynchronous Reinforcement Learning Framework for Vision-Language-Action Models
图 1 · 摘自论文原文
  • 通过物理解耦训练数据与权重控制,分离仿真与优化干扰。
  • 四线程异步流水线实现采样、推理、梯度计算与参数分发全重叠。
  • 适用于百亿参数以上视觉语言动作模型的高并发分布式训练,适合通用智能体研发。

具身人工智能的快速发展使视觉-语言-动作(VLA)模型在多模态感知与任务执行方面表现卓越。然而,在大规模分布式环境中对这些巨型模型应用强化学习(RL)时,面临严重系统瓶颈,主要源于高保真物理仿真与深度学习对显存/带宽的高强度需求之间的资源冲突,导致整体吞吐量受限于执行阶段的低效。为此,我们提出 D-VLA,一个面向大规模具身基础模型的高并发、低延迟分布式强化学习框架。D-VLA 引入“平面解耦”机制,物理隔离高频训练数据与低频权重控制,消除仿真与优化间的干扰。进一步设计四线程异步“泳道”流水线,实现采样、推理、梯度计算与参数分发的完全并行重叠。同时,采用双池显存管理模型与拓扑感知复制策略,缓解内存碎片并优化通信效率。在 LIBERO 等基准测试中,D-VLA 显著优于主流 RL 框架,在百亿参数 VLA 模型上提升吞吐量与采样效率;在万亿参数可扩展性测试中,框架保持优异稳定性与线性加速能力,为高性能通用具身智能体提供坚实系统支持。

原文摘要 · Abstract (English)

The rapid evolution of Embodied AI has enabled Vision-Language-Action (VLA) models to excel in multimodal perception and task execution. However, applying Reinforcement Learning (RL) to these massive models in large-scale distributed environments faces severe systemic bottlenecks, primarily due to the resource conflict between high-fidelity physical simulation and the intensive VRAM/bandwidth demands of deep learning. This conflict often leaves overall throughput constrained by execution-phase inefficiencies. To address these challenges, we propose D-VLA, a high-concurrency, low-latency distributed RL framework for large-scale embodied foundation models. D-VLA introduces "Plane Decoupling," physically isolating high-frequency training data from low-frequency weight control to eliminate interference between simulation and optimization. We further design a four-thread asynchronous "Swimlane" pipeline, enabling full parallel overlap of sampling, inference, gradient computation, and parameter distribution. Additionally, a dual-pool VRAM management model and topology-aware replication resolve memory fragmentation and optimize communication efficiency. Experiments on benchmarks like LIBERO show that D-VLA significantly outperforms mainstream RL frameworks in throughput and sampling efficiency for billion-parameter VLA models. In trillion-parameter scalability tests, our framework maintains exceptional stability and linear speedup, providing a robust system for high-performance general-purpose embodied agents.

强化学习分布式训练视觉语言动作高并发

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。