用积木堆叠任务训练大模型的空间推理能力
SpatialBlock: Enhancing Spatial Intelligence in LVLMs via Synthetic Block-Stacking Problem
- 通过合成积木堆叠任务提升视觉语言模型的空间理解力
- 在1.5万条合成数据上训练后，模型空间推理能力显著超越基线
- 适合关注3D结构理解、少标注场景下的模型训练者
大型视觉语言模型在多种视觉任务中表现优异，但在重构和推理2D图像中场景的3D结构——即空间智能——方面仍有限。现有方法依赖需密集几何标注的真实场景空间问答数据集，但构建此类标签成本高、耗时长，且常因依赖外部感知模块而引入噪声。本文受人类认知发展启发，提出一种新范式：通过结构化的积木操作任务学习基础空间技能。我们构建了SpatialBlock-15k数据集，包含1.5万条积木堆叠问题，涵盖3D到2D投影、视角变换及结构组合。数据集还引入可控颜色调制作为视觉线索，以促进复杂视觉条件下基于锚点的推理。实验表明，无论采用直接回答还是基于推理的预测方式，模型在该数据集上训练后，均显著优于基线，并能泛化至真实世界空间任务，即便数据为合成且紧凑。代码与数据见https://github.com/rsoohyun/SpatialBlock。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have achieved strong performance on diverse visual tasks, yet their ability to reconstruct and reason about the 3D structure of the scene depicted in 2D images -- referred to as spatial intelligence -- remains limited. Existing approaches attempt to address this gap by using real-scene spatial question answering datasets that require dense geometric annotations. However, constructing such labels is costly, time-consuming, and often noisy due to reliance on external perception modules. In this work, we propose a novel paradigm inspired by human cognitive development: learning foundational spatial skills through structured block-manipulation tasks. We introduce SpatialBlock-15k, a synthetic dataset of 15,000 block-stacking problems covering 3D-to-2D projection, viewpoint transformation, and structural combination. The dataset further incorporates controlled color modulation as visual cues to encourage anchor-based reasoning in visually complex conditions. Experiments demonstrate that LVLMs trained on our dataset through either direct answering or reasoning-based prediction significantly outperform baselines and generalize to real-world spatial tasks, despite the dataset's synthetic and compact nature. Code and data are available at https://github.com/rsoohyun/SpatialBlock.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。