测试大模型能否仅凭文字描述理解空间关系,发现其有潜在空间推理能力。
Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptions

- 将视觉场景转为带坐标的文字描述,让大模型做符号推理而非看图
- 顶尖模型在局部任务表现好,但全局一致性仍有明显差距
- 显式空间推理能显著提升性能,适合研究具身智能与视觉语言模型
近期空间智能(SI)研究主要依赖视觉语言模型(VLMs),但一个关键问题仍未解决:空间理解来自视觉编码器,还是模型本身的推理能力?为此,我们提出SiT-Bench,一个全新的基准,用于评估大语言模型(LLMs)在无像素输入下的空间智能表现。该基准包含超过3,800个专家标注的项目,涵盖五大类共17个子任务,从第一人称导航、视角变换到精细机器人操作。通过将单/多视角场景转化为高保真、坐标感知的文本描述,我们挑战LLMs进行符号化文本推理,而非视觉模式匹配。对当前SOTA LLMs的评估显示,模型在局部语义任务上表现良好,但在全局一致性方面存在显著“空间鸿沟”。值得注意的是,显式空间推理能显著提升性能,表明LLMs具备潜在的世界建模能力。所提出的SiT-Bench数据集将成为未来具身智能与空间感知语言模型发展的基础资源。代码与数据集将开源于https://github.com/binisalegend/SiT-Bench。
原文摘要 · Abstract (English)
Recent advancements in Spatial Intelligence (SI) have predominantly relied on Vision-Language Models (VLMs), yet a critical question remains: does spatial understanding originate from visual encoders or the fundamental reasoning backbone? Inspired by this question, we introduce SiT-Bench, a novel benchmark designed to evaluate the SI performance of Large Language Models (LLMs) without pixel-level input, comprises over 3,800 expert-annotated items across five primary categories and 17 subtasks, ranging from egocentric navigation and perspective transformation to fine-grained robotic manipulation. By converting single/multi-view scenes into high-fidelity, coordinate-aware textual descriptions, we challenge LLMs to perform symbolic textual reasoning rather than visual pattern matching. Evaluation results of state-of-the-art (SOTA) LLMs reveals that while models achieve proficiency in localized semantic tasks, a significant "spatial gap" remains in global consistency. Notably, we find that explicit spatial reasoning significantly boosts performance, suggesting that LLMs possess latent world-modeling potential. Our proposed dataset SiT-Bench serves as a foundational resource to foster the development of spatially-grounded LLM backbones for future VLMs and embodied agents. Our code and benchmark will be released at https://github.com/binisalegend/SiT-Bench .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。