用自然语言表示动作,避免视觉语言模型在机器人训练中遗忘原有能力。
Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgetting
- 将低层动作转为自然语言,解决数据分布不匹配问题。
- 仅用低秩适配(LoRA)微调,保留原始模型的推理与理解能力。
- 支持零样本泛化,适用于多语言指令和开放世界任务。
将视觉语言模型(VLM)在机器人遥操作数据上微调以构建视觉-语言-动作(VLA)模型,是训练通用策略的有前景方法,但面临根本性权衡:学习生成动作常削弱VLM的基础推理与多模态理解能力,影响对新场景、指令遵循和语义理解的泛化性能。我们认为这种灾难性遗忘源于VLM互联网级预训练语料与机器人微调数据之间的分布不匹配。受此启发,我们提出VLM2VLA:一种先在数据层面通过自然语言表示低层动作来对齐分布的VLA训练范式。该对齐使仅使用低秩适配(LoRA)即可训练VLAs,从而最小化对VLM主干的修改,避免灾难性遗忘。结果表明,无需改变底层架构且无需在互联网规模VLM数据集上联合训练,即可在机器人遥操作数据上微调VLM。通过大量视觉问答(VQA)研究及800余次真实机器人实验,我们证明VLM2VLA能保持VLM核心能力,实现对需开放世界语义推理与多语言指令遵循的新任务的零样本泛化。
原文摘要 · Abstract (English)
Fine-tuning vision-language models (VLMs) on robot teleoperation data to create vision-language-action (VLA) models is a promising paradigm for training generalist policies, but it suffers from a fundamental tradeoff: learning to produce actions often diminishes the VLM's foundational reasoning and multimodal understanding, hindering generalization to novel scenarios, instruction following, and semantic understanding. We argue that this catastrophic forgetting is due to a distribution mismatch between the VLM's internet-scale pretraining corpus and the robotics fine-tuning data. Inspired by this observation, we introduce VLM2VLA: a VLA training paradigm that first resolves this mismatch at the data level by representing low-level actions with natural language. This alignment makes it possible to train VLAs solely with Low-Rank Adaptation (LoRA), thereby minimally modifying the VLM backbone and averting catastrophic forgetting. As a result, the VLM can be fine-tuned on robot teleoperation data without fundamentally altering the underlying architecture and without expensive co-training on internet-scale VLM datasets. Through extensive Visual Question Answering (VQA) studies and over 800 real-world robotics experiments, we demonstrate that VLM2VLA preserves the VLM's core capabilities, enabling zero-shot generalization to novel tasks that require open-world semantic reasoning and multilingual instruction following.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。