探索视觉语言模型如何更好初始化机器人决策系统。
Rethinking VLM Representation for VLA Initialization
- 通过三维度实验研究预训练表示的适配方法。
- 原生视觉语言表征对动作性能贡献关键，但微调效果依赖下游任务瓶颈。
- 分阶段LoRA训练能有效保留预训练能力并注入机器人轨迹信号。
视觉-语言-动作（VLA）模型普遍采用预训练视觉-语言模型（VLM）作为策略主干，但何种预训练表征适合用于VLA初始化仍不明确。本文将VLA初始化视为受控的表征设计问题，沿三个维度展开研究：具身视觉问答（embodied VQA）监督、参数更新策略和机器人数据预训练。实验表明，原始预训练VLM表征是动作性能的关键来源。然而，具身VQA适应并非带来统一增益，其效果取决于下游瓶颈，且不同能力域的收益不可简单叠加。在更新策略上，LoRA相比全量微调提供更可靠的初始化，表明过度重塑预训练表征会削弱其对动作学习的贡献。机器人数据预训练进一步提升初始化效果，最优方案为分阶段基于LoRA的训练。结果表明，有效的VLM到VLA迁移应注入与动作相关的具身和机器人轨迹信号，同时保留对动作学习仍有价值的预训练表征。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) models widely adopt pretrained Vision-Language Models (VLMs) as policy backbones, yet it remains unclear what kind of pretrained VLM representation is useful as a VLA initialization. In this paper, we study VLA initialization as a controlled representation-design problem along three axes: capability-level embodied VQA supervision, parameter-update strategy, and robot-data pretraining. Our experiments show that the original pretrained VLM representation is a key source of action performance. However, embodied VQA adaptation does not yield uniform gains: its benefit depends on downstream bottlenecks, and gains from different capability domains are not simply additive. For update strategy, LoRA provides a more reliable initialization than Full Finetune, indicating that overly reshaping the pretrained representation can weaken VLA initialization. Robot-data pretraining further improves VLA initialization, with the strongest variant obtained by staged LoRA-based training. Together, these findings suggest that effective VLM-to-VLA adaptation should inject action-relevant embodied and robot-trajectory signals while preserving the pretrained VLM representation that remains useful for action learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。