arXiv:2606.19297cs.LGcs.RO2026-06

测试视觉语言动作模型对常识和世界知识的保留能力,发现其在复杂概念上表现明显下降。

Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models

论文配图:Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models
图 1 · 摘自论文原文
  • 设计新评测协议Act2Answer,用动作选择答案来减少控制干扰
  • 7个VLA模型在丰富语义类别上比源VLM差30%以上
  • 中层网络最擅长提取答案相关信号,适合研究模型知识保留机制

具身视觉-语言-动作(VLA)模型通常通过在机器人数据上微调强大的预训练视觉语言模型(VLM)获得,但其在适应后仍保留多少常识和事实知识尚不明确。知识敏感任务中的失败可能源于知识缺失或低级控制泛化能力差。我们提出Act2Answer,一种轻量级评估协议,将VLM知识基准适配为VLA评估,要求代理通过动作回答问题。每个问题转化为一个短桌面试验,代理执行单次物体放置动作以选择候选答案,从而获得去除了控制混淆的动作基线成功率。我们构建了涵盖多样常识与世界知识类别的测试环境,并引入分层意图探测以定位答案相关信息在VLM主干和动作头中的分布。对7个VLA模型和9个VLM基线的大规模研究显示,尽管在简单概念上表现良好,但VLA在丰富语义类别上的表现显著落后于其源模型;与视觉问答(VQA)联合训练相关的模型知识保留更好;答案相关信号在中层网络达到峰值,但在高层逐渐减弱。Act2Answer代码已开源。

原文摘要 · Abstract (English)

Embodied Vision-Language-Action (VLA) models are typically obtained by fine-tuning powerful pretrained VLMs on robotics data, yet it is unclear how much commonsense and factual knowledge they retain after adaptation. Failures on knowledge-sensitive tasks are ambiguous, conflating missing knowledge with poor generalization of low-level control. We introduce Act2Answer, a lightweight protocol that adapts VLM knowledge benchmarks to VLA evaluation by requiring agents to answer through action. Each question becomes a short tabletop episode where the agent performs a single object-placement action to select among candidate answers, yielding an action-grounded success rate with reduced control confounds. We curate a test suite of such environments across diverse commonsense and world-knowledge categories and introduce layerwise intent probing to localize answer-relevant information across the VLM backbone and action head. In a large-scale study of 7 VLA models and 9 VLM baselines, we systematically rank models across categories, finding that VLAs show solid performance on simple concepts while exhibiting larger gaps on richer semantic categories relative to their source VLMs, that VQA co-training is associated with better knowledge retention, and that answer-relevant signals peak in middle VLA layers but attenuate in upper layers. Act2Answer is available at https://tttonyalpha.github.io/act2answer/.

VLA模型常识推理知识保留评测方法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。