让视觉语言模型在机器人控制中更快更准,同时保住原有知识。
Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better

- 用隔离技术保护预训练模型,避免新模块干扰其语义能力。
- 实验表明,不加隔离会显著拖慢训练速度并削弱知识迁移。
- 适合关注高效机器人控制与模型知识保持的研究者。
视觉-语言-动作(VLA)模型通过结合端到端学习与大规模视觉语言模型(VLM)的语义知识迁移,为物理系统(如机器人)训练控制策略提供了强大方法。然而,实时控制需求与VLM设计存在矛盾:高性能VLM通常拥有数十亿至数百亿参数,难以实现实时推理,且输出为离散标记而非机器人所需的连续值。为此,现有VLA模型引入了专用模块(如动作专家或连续输出头),但这些模块常需新增未训练参数,可能影响预训练模型的语义知识。本文研究包含连续扩散或流匹配动作专家的VLA模型,发现直接加入此类专家会严重损害训练速度与知识转移。通过系统分析多种设计选择及其对性能和知识保留的影响,提出一种在训练过程中隔离VLM主干的方法,有效缓解该问题。视频演示见 https://pi.website/research/knowledge_insulation。
原文摘要 · Abstract (English)
Vision-language-action (VLA) models provide a powerful approach to training control policies for physical systems, such as robots, by combining end-to-end learning with transfer of semantic knowledge from web-scale vision-language model (VLM) training. However, the constraints of real-time control are often at odds with the design of VLMs: the most powerful VLMs have tens or hundreds of billions of parameters, presenting an obstacle to real-time inference, and operate on discrete tokens rather than the continuous-valued outputs that are required for controlling robots. To address this challenge, recent VLA models have used specialized modules for efficient continuous control, such as action experts or continuous output heads, which typically require adding new untrained parameters to the pretrained VLM backbone. While these modules improve real-time and control capabilities, it remains an open question whether they preserve or degrade the semantic knowledge contained in the pretrained VLM, and what effect they have on the VLA training dynamics. In this paper, we study this question in the context of VLAs that include a continuous diffusion or flow matching action expert, showing that naively including such experts significantly harms both training speed and knowledge transfer. We provide an extensive analysis of various design choices, their impact on performance and knowledge transfer, and propose a technique for insulating the VLM backbone during VLA training that mitigates this issue. Videos are available at https://pi.website/research/knowledge_insulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。