arXiv:2410.16162cs.CVcs.CL2024-10EMNLP被引 29

通过训练模型掌握基础空间能力,显著提升视觉语言模型的空间推理能力。

Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoning

  • 将二维空间推理拆解为方向、距离、定位三类基础能力进行专项训练。
  • 在合成数据上训练后,模型在复杂和真实场景任务中准确率显著提升。
  • 适合需要强空间理解的机器人导航、交互系统等应用开发者使用。

视觉语言模型(VLMs)在众多任务上表现良好,但在空间推理方面常出现错误,而空间推理对导航与物理环境交互至关重要。我们的评估发现,当前最先进的VLMs在复合空间问题(如人类轻易解决的路径规划)上给出不合理或错误答案。为此,我们提出Sparkle框架,仅通过训练基本空间能力来增强VLM的二维空间推理能力。首先将2D空间推理分解为方向理解、距离估计和定位三个核心组件,并假设掌握这些基础技能可显著提升复杂空间任务的表现及泛化能力。Sparkle通过生成合成数据,针对这三项能力提供精准监督,构建每个能力的指令数据集。实验表明,经Sparkle微调的VLM不仅在基础任务上表现更好,还在复合任务和分布外的真实世界空间推理任务中取得显著提升。结果证明,通过合成泛化强化基础空间技能,能有效推进复杂空间推理,为提升VLM空间理解提供了系统性策略。源代码已公开于https://github.com/YihongT/Sparkle。

原文摘要 · Abstract (English)

Vision language models (VLMs) perform well on many tasks but often fail at spatial reasoning, which is essential for navigation and interaction with physical environments. Many spatial reasoning tasks depend on fundamental two-dimensional (2D) skills, yet our evaluation shows that state-of-the-art VLMs give implausible or incorrect answers to composite spatial problems, including simple pathfinding tasks that humans solve effortlessly. To address this, we enhance 2D spatial reasoning in VLMs by training them only on basic spatial capabilities. We first disentangle 2D spatial reasoning into three core components: direction comprehension, distance estimation, and localization. We hypothesize that mastering these skills substantially improves performance on complex spatial tasks that require advanced reasoning and combinatorial problem solving, while also generalizing to real-world scenarios. To test this, we introduce Sparkle, a framework that generates synthetic data to provide targeted supervision across these three capabilities and yields an instruction dataset for each. Experiments show that VLMs fine-tuned with \emph{Sparkle} improve not only on basic tasks but also on composite and out-of-distribution real-world spatial reasoning tasks. These results indicate that enhancing basic spatial skills through synthetic generalization effectively advances complex spatial reasoning and offers a systematic strategy for boosting the spatial understanding of VLMs. Source codes of Sparkle are available at https://github.com/YihongT/Sparkle.

空间推理视觉语言模型合成数据泛化能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。