用小模型教大模型做动作,效率更高成功率也更高。
VITA-VLA: Efficiently Teaching Vision-Language Models to Act via Action Expert Distillation
- 用小模型知识蒸馏,让视觉语言模型学会执行动作
- 在LIBERO数据集上成功率达97.3%,比之前方法提升11.8%
- 真实机器人实验中成功率提升17%,训练成本大幅降低
视觉-语言-动作(VLA)模型通过融合预训练视觉-语言模型(VLM)的感知能力,显著提升了机器人操作性能。然而从头训练成本高昂。本文提出一种基于知识蒸馏的轻量级框架,通过将小型动作模型的知识迁移到VLM中,赋予其动作执行能力。架构仅新增动作标记和状态编码器,保留原VLM结构。采用两阶段训练:先轻量对齐,将VLM隐藏状态映射至小模型动作空间,复用其预训练解码器;再选择性微调语言模型、状态编码器和动作模块,实现多模态输入融合与精准动作生成。动作标记使模型可直接预测未来动作,状态编码器则引入视觉未覆盖的机器人动力学信息。该设计显著降低训练开销。相比现有最优方法,本方法在LIBERO上平均成功率提升至97.3%(+11.8%),在LIBERO-LONG上达93.5%(+24.5%)。真实世界五项任务中,成功率82.0%(+17%),证明动作蒸馏有效提升动作精度并大幅降低训练成本。
原文摘要 · Abstract (English)
Vision-Language Action (VLA) models significantly advance robotic manipulation by leveraging the strong perception capabilities of pretrained vision-language models (VLMs). By integrating action modules into these pretrained models, VLA methods exhibit improved generalization. However, training them from scratch is costly. In this work, we propose a simple yet effective distillation-based framework that equips VLMs with action-execution capability by transferring knowledge from pretrained small action models. Our architecture retains the original VLM structure, adding only an action token and a state encoder to incorporate physical inputs. To distill action knowledge, we adopt a two-stage training strategy. First, we perform lightweight alignment by mapping VLM hidden states into the action space of the small action model, enabling effective reuse of its pretrained action decoder and avoiding expensive pretraining. Second, we selectively fine-tune the language model, state encoder, and action modules, enabling the system to integrate multimodal inputs with precise action generation. Specifically, the action token provides the VLM with a direct handle for predicting future actions, while the state encoder allows the model to incorporate robot dynamics not captured by vision alone. This design yields substantial efficiency gains over training large VLA models from scratch. Compared with previous state-of-the-art methods, our method achieves 97.3% average success rate on LIBERO (11.8% improvement) and 93.5% on LIBERO-LONG (24.5% improvement). In real-world experiments across five manipulation tasks, our method consistently outperforms the teacher model, achieving 82.0% success rate (17% improvement), which demonstrate that action distillation effectively enables VLMs to generate precise actions while substantially reducing training costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。