探究视觉语言模型空间推理能力受限的根源，发现编码器设计与位置编码方式是关键瓶颈。
Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA
- 对比不同图像编码器和位置编码方式，揭示其对空间理解的影响。
- 在多个空间基准测试中，现有模型仍存在显著性能差距。
- 适合关注视觉语言模型设计缺陷与改进方向的研究者阅读。
视觉语言模型（VLMs）虽快速进步，但在基本的空间推理上仍表现脆弱，尤其在相对位置、布局和计数等二维空间关系理解上。我们认为这种失败并非仅由数据不足导致，而是与当前主流VLM架构的设计选择密切相关：依赖CLIP式图像编码器，以及将图像展平为一维标记序列并使用一维位置编码。本文在LLaVA框架内开展受控诊断研究，以分离这些设计因素对空间定位的影响。我们在一系列空间基准测试上评估前沿模型及LLaVA变体，比较基于CLIP的编码器与采用更密集或生成式目标训练的替代方案，以及引入二维位置编码的变体。结果表明，各模型间存在持续的空间性能差距，且编码器目标与位置结构确实影响空间行为，但尚未完全解决该问题。
原文摘要 · Abstract (English)
Vision-language models (VLMs) have advanced rapidly, yet they still struggle with basic spatial reasoning. Despite strong performance on general benchmarks, modern VLMs remain brittle at understanding 2D spatial relationships such as relative position, layout, and counting. We argue that this failure is not merely a data problem, but is closely tied to dominant design choices in current VLM pipelines: reliance on CLIP-style image encoders and the flattening of images into 1D token sequences with 1D positional encoding. We present a controlled diagnostic study within the LLaVA framework to isolate how these choices affect spatial grounding. We evaluate frontier models and LLaVA variants on a suite of spatial benchmarks, comparing CLIP-based encoders against alternatives trained with denser or generative objectives, as well as variants augmented with 2D positional encoding. Our results show consistent spatial performance gaps across models, and indicate that encoder objectives and positional structure shape spatial behavior, but do not fully resolve it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。