测试视觉语言模型是否理解3D空间布局，发现它们只擅长物体识别。
Do Vision-Language Models Understand 3D Scenes or Just Catalogue Objects?
- 构建3034样本基准，测试深度遮挡、反射推理和体积重排三类空间理解能力。
- 模型在重排任务准确率达53%-97%，但遮挡测试仅6%-45%，反射测试低于7%。
- 失败根源在视觉-语言令牌融合环节，空间信息被压缩后丢失。
视觉语言模型能可靠命名场景中的物体，但它们是否理解物体所处的3D布局？我们引入一个包含3,034个样本的人工标注基准，聚焦三个空间理解维度：深度顺序遮挡（通过三种独立反事实操作检验）、可见反射的光学-几何推理，以及体积重排规划。六种前沿及开源视觉语言模型由受训标注员评估，共生成18,204条回答，全程无LLM作为裁判。结果揭示显著脱节：能在可见布局上实现53%-97%准确率的重排规划且极少违反碰撞约束的模型，在遮挡测试中表现降至6%-45%，反射推理则低于7%。具身推理模型也表现出相同模式。对Qwen3-VL-8B-Thinking的白盒分析显示，失败源于视觉-语言令牌融合：空间信息在视觉编码器中可恢复，但在令牌压缩后不可访问，仅在向语言解码器注入干净后融合激活时才重新稳定。
原文摘要 · Abstract (English)
Vision-language models reliably name objects in a scene, but do they represent the 3D layout those objects inhabit? We introduce a 3,034-sample human-curated benchmark targeting three components of spatial understanding: depth-ordered occlusion (probed via three independent counterfactual operationalisations), optical-geometry inference over visible reflections, and volumetric rearrangement planning. Six frontier and open-weight VLMs, scored by trained annotators on 18,204 responses with no LLM-as-judge, reveal a sharp dissociation: models that plan rearrangements over visible layouts at 53--97% accuracy and rarely violate collision constraints fall to 6--45% on occlusion and below 7% on reflections. An embodied-reasoning model reproduces the same profile. White-box analysis on Qwen3-VL-8B-Thinking localises the failure to the visual-token merger: spatial information recoverable throughout the vision encoder becomes inaccessible after token compression and only stabilises again when clean post-merger activations are patched into the language decoder.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。