提出双流结构UAM,让视觉语言动作模型不丢语义能力。
UAM: A Dual-Stream Perspective on Forgetting in VLA Training

- 用双流架构分离语义与控制路径,避免单编码器负担过重。
- 训练后保留95%以上原始视觉语言模型的多模态能力。
- 无需冻结参数或辅助训练,适合需要泛化的机器人任务。
视觉-语言-动作(VLA)模型通常在动作数据上微调预训练的视觉-语言模型(VLM),但我们发现这种标准方法会系统性削弱VLM的多模态能力,称为‘具身税’。是否必须遗忘?受生物视觉双通路启发,我们发现当前VLA将语言语义与控制相关视觉特征压缩于单一编码器,形成结构性瓶颈。为此,我们提出统一动作模型(UAM),引入并行的背侧专家(Dorsal Expert),类比大脑背侧通路。该专家由预训练生成模型初始化,并通过预测视觉动态的中层推理目标进行训练,减轻对VLM的控制学习压力。整个VLA可仅用动作数据端到端训练:无需参数冻结、梯度截断或辅助的VL共训练。实验表明,UAM在多种操作任务中保持超过95%的原始VLM多模态能力,并在未见物体、新物体-目标组合及指令变化等分布外泛化测试中取得最优平均成功率。结果表明,语义保留在架构分离中自然产生,而非依赖冻结权重或辅助数据重播,且此能力可自然迁移至动作的语义泛化。
原文摘要 · Abstract (English)
Vision--language--action (VLA) models are typically built by fine-tuning a pretrained vision--language model (VLM) on action data. However, we show that this standard recipe systematically erodes the VLM's multimodal competence, a side effect we call the embodiment tax. But do VLAs have to forget? Inspired by the two-stream organization of biological vision, we trace this degradation to a structural bottleneck: current VLAs ask a single encoder to support both language-grounded semantics and control-relevant visual features, whereas biological vision separates recognition and visuomotor control into distinct pathways. Building on this view, we propose the Unified Action Model (UAM), which adds a parallel Dorsal Expert, an analog of the brain's dorsal pathway. To make the Dorsal Expert an effective second pathway and reduce the control-learning burden on the VLM, we initialize it from a pretrained generative model and train it with a mid-level reasoning objective that predicts visual dynamics. This design allows us to train the whole VLA end-to-end on action data alone: with no parameter freezing, no gradient stopping, and no auxiliary VL co-training, UAM retains over $95\%$ of the underlying VLM's multimodal capability and at the same time achieves the highest average success rate among baselines on a variety of manipulation tasks that probe out-of-distribution generalization, including unseen objects, novel object--target compositions, and instruction variation. Together, these results suggest that semantic preservation in VLAs can emerge from architectural separation itself, rather than being enforced by frozen weights or auxiliary data replay, and that this preserved semantic capability can naturally transfer from VLMs to semantic generalization in actions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。