测试视觉语言模型的空间理解能力，发现其在方向与旋转任务上表现远差于人类。
LRR-Bench: Left, Right or Rotate? Vision-Language models Still Struggle With Spatial Understanding Tasks
- 构建合成数据集，区分绝对位置与三维空间运动两类空间理解任务。
- 顶尖模型在复杂任务上仅达人类性能的10%左右，部分任务接近零分。
- 适合关注多模态模型空间推理短板的研究者和机器人应用开发者。
真实世界应用如自动驾驶与人形机器人操作需要精确的空间感知能力。然而，视觉语言模型（VLMs）如何识别空间关系与感知空间运动仍缺乏深入研究。本文提出一种空间评估流程，并构建相应基准数据集LRR-Bench。将空间理解分为两类：绝对空间理解（如物体在图像中的左右位置），以及三维空间理解（包括移动与旋转）。该数据集完全为合成数据，可低成本生成测试样本并避免数据污染。我们在多个前沿VLM上进行实验，发现人类在所有任务中表现接近完美，而当前VLM仅在两个最简单任务上达到人类水平；其余任务中，模型表现显著低于人类，甚至在多个任务上得分接近零。代码与数据集已开源。
原文摘要 · Abstract (English)
Real-world applications, such as autonomous driving and humanoid robot manipulation, require precise spatial perception. However, it remains underexplored how Vision-Language Models (VLMs) recognize spatial relationships and perceive spatial movement. In this work, we introduce a spatial evaluation pipeline and construct a corresponding benchmark. Specifically, we categorize spatial understanding into two main types: absolute spatial understanding, which involves querying the absolute spatial position (e.g., left, right) of an object within an image, and 3D spatial understanding, which includes movement and rotation. Notably, our dataset is entirely synthetic, enabling the generation of test samples at a low cost while also preventing dataset contamination. We conduct experiments on multiple state-of-the-art VLMs and observe that there is significant room for improvement in their spatial understanding abilities. Explicitly, in our experiments, humans achieve near-perfect performance on all tasks, whereas current VLMs attain human-level performance only on the two simplest tasks. For the remaining tasks, the performance of VLMs is distinctly lower than that of humans. In fact, the best-performing Vision-Language Models even achieve near-zero scores on multiple tasks. The dataset and code are available on https://github.com/kong13661/LRR-Bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。