arXiv:2601.03309cs.CVcs.AI2026-01被引 31

探究视觉语言模型如何影响智能体动作决策,发现视觉模块是关键瓶颈。

VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models

  • 用极简参数将通用视觉语言模型转为动作策略,实现公平对比。
  • 通用能力强的模型不一定在具体任务中表现好,需针对性优化视觉模块。
  • 在视觉编码器中注入控制相关监督,即使冻结也能提升性能,适合机器人研究者。

视觉-语言-动作(VLA)模型通过将预训练的大规模视觉语言模型(VLM)集成到策略主干中,展现出出色的泛化能力。本文重新审视一个基本却少有系统研究的问题:VLM的选择与能力如何影响下游VLA策略的表现?我们提出VLM4VLA,一种仅需少量可学习参数的最小化适配流程,可将通用VLM转化为VLA策略,实现公平高效的比较。尽管结构简单,其性能仍媲美更复杂的网络设计。在三个基准上的广泛实验证明,使用VLM初始化相比从零训练有稳定优势,但通用能力并非下游任务表现的良好预测因子。这挑战了常见假设:标准VLM能力是必要但不充分条件。进一步通过在七个辅助具身任务(如具身问答、视觉指认、深度估计)上微调VLM,发现提升特定具身技能表现并不保证下游控制性能提升。模态级消融分析表明,视觉模块而非语言部分是主要性能瓶颈。实验显示,向VLM的视觉编码器注入控制相关监督,即使在下游微调时保持冻结,也能带来持续增益,揭示当前VLM预训练目标与具身动作规划需求间存在持久的领域差距。

原文摘要 · Abstract (English)

Vision-Language-Action (VLA) models, which integrate pretrained large Vision-Language Models (VLM) into their policy backbone, are gaining significant attention for their promising generalization capabilities. This paper revisits a fundamental yet seldom systematically studied question: how VLM choice and competence translate to downstream VLA policies performance? We introduce VLM4VLA, a minimal adaptation pipeline that converts general-purpose VLMs into VLA policies using only a small set of new learnable parameters for fair and efficient comparison. Despite its simplicity, VLM4VLA proves surprisingly competitive with more sophisticated network designs. Through extensive empirical studies on various downstream tasks across three benchmarks, we find that while VLM initialization offers a consistent benefit over training from scratch, a VLM's general capabilities are poor predictors of its downstream task performance. This challenges common assumptions, indicating that standard VLM competence is necessary but insufficient for effective embodied control. We further investigate the impact of specific embodied capabilities by fine-tuning VLMs on seven auxiliary embodied tasks (e.g., embodied QA, visual pointing, depth estimation). Contrary to intuition, improving a VLM's performance on specific embodied skills does not guarantee better downstream control performance. Finally, modality-level ablations identify the visual module in VLM, rather than the language component, as the primary performance bottleneck. We demonstrate that injecting control-relevant supervision into the vision encoder of the VLM yields consistent gains, even when the encoder remains frozen during downstream fine-tuning. This isolates a persistent domain gap between current VLM pretraining objectives and the requirements of embodied action-planning.

视觉语言模型具身智能动作策略多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。