arXiv:2604.19728cs.ROcs.AI2026-04被引 1

统一训练视觉-语言-动作模型,支持从零到多任务部署

VLA Foundry: A Unified Framework for Training Vision-Language-Action Models

论文配图:VLA Foundry: A Unified Framework for Training Vision-Language-Action Models
图 1 · 摘自论文原文
  • 一套代码实现大语言、视觉语言到动作模型的全流程训练
  • 自研模型在模拟器上表现媲美闭源成果,基于Qwen3-VL更强
  • 开源框架与权重,适合机器人多任务学习研究者使用

我们提出VLA Foundry,一个开源框架,统一大语言模型、视觉语言模型和视觉-语言-动作模型的训练流程。现有开源工作多聚焦于动作训练阶段,常拼接不兼容的预训练管道。VLA Foundry提供共享训练栈,支持从语言预训练到动作专家微调的端到端控制。框架支持从零训练和基于Hugging Face的预训练主干。为验证其有效性,我们训练并发布了两类模型:第一类通过LLM→VLM→VLA管道从零训练;第二类基于预训练的Qwen3-VL主干构建。我们在开放数据、开源的LBM Eval模拟器上评估闭环策略性能,并改进了模拟器和STEP分析工具以提升可复用性。在标准评估设置下,我们的完全开源自研模型表现媲美先前闭源工作;采用Qwen3-VL主干后,多任务桌面操作策略显著优于基线。VLA Foundry代码库位于https://github.com/TRI-ML/vla_foundry,所有多任务模型权重已发布于https://huggingface.co/collections/TRI-ML/vla-foundry。更多可视化视频见项目官网https://tri-ml.github.io/vla_foundry。

原文摘要 · Abstract (English)

We present VLA Foundry, an open-source framework that unifies LLM, VLM, and VLA training in a single codebase. Most open-source VLA efforts specialize on the action training stage, often stitching together incompatible pretraining pipelines. VLA Foundry instead provides a shared training stack with end-to-end control, from language pretraining to action-expert fine-tuning. VLA Foundry supports both from-scratch training and pretrained backbones from Hugging Face. To demonstrate the utility of our framework, we train and release two types of models: the first trained fully from scratch through our LLM-->VLM-->VLA pipeline and the second built on the pretrained Qwen3-VL backbone. We evaluate closed-loop policy performance of both models on LBM Eval, an open-data, open-source simulator. We also contribute usability improvements to the simulator and the STEP analysis tools for easier public use. In the nominal evaluation setting, our fully-open from-scratch model is on par with our prior closed-source work and substituting in the Qwen3-VL backbone leads to a strong multi-task table top manipulation policy outperforming our baseline by a wide margin. The VLA Foundry codebase is available at https://github.com/TRI-ML/vla_foundry and all multi-task model weights are released on https://huggingface.co/collections/TRI-ML/vla-foundry. Additional qualitative videos are available on the project website https://tri-ml.github.io/vla_foundry.

视觉-语言-动作机器人学习开源框架多任务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。