探究视觉语言模型能否零样本识别物体状态，发现现有模型表现不佳。
Do Pre-trained Vision-Language Models Encode Object States?
- 构建新数据集ChangeIt-Frames，用零样本提示测试模型对物体状态的识别能力。
- 九个主流VLM在物体状态区分上普遍失败，准确率低于随机水平。
- 提出三大改进方向：定位精度、概念-物体绑定架构、判别式编码目标。
为使视觉语言模型（VLM）理解物理世界（如因果关系），需先捕捉视觉世界的时序动态，例如物体状态随时间的变化（如整苹果变为切片苹果）。本文旨在探究预训练于网络规模数据的VLM是否能通过零样本文本提示提取物体状态信息。我们构建了物体状态识别数据集ChangeIt-Frames，评估了九个开源VLM，涵盖对比学习与生成式训练模型。实验表明，尽管这些先进模型在物体识别上表现可靠，却始终无法准确区分物体的物理状态。通过大量实验，我们识别出三个关键改进方向：物体定位质量、概念与物体绑定的架构设计、以及针对物体状态学习判别性视觉与语言编码器的目标。数据与代码已公开。
原文摘要 · Abstract (English)
For a vision-language model (VLM) to understand the physical world, such as cause and effect, a first step is to capture the temporal dynamics of the visual world, for example how the physical states of objects evolve over time (e.g. a whole apple into a sliced apple). Our paper aims to investigate if VLMs pre-trained on web-scale data learn to encode object states, which can be extracted with zero-shot text prompts. We curate an object state recognition dataset ChangeIt-Frames, and evaluate nine open-source VLMs, including models trained with contrastive and generative objectives. We observe that while these state-of-the-art vision-language models can reliably perform object recognition, they consistently fail to accurately distinguish the objects' physical states. Through extensive experiments, we identify three areas for improvements for VLMs to better encode object states, namely the quality of object localization, the architecture to bind concepts to objects, and the objective to learn discriminative visual and language encoders on object states. Data and code are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。