arXiv:2505.12448cs.CV2025-05NeurIPS被引 39

用深度图生成可解释文本推理,提升视觉语言模型的空间理解能力

SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning

  • 将深度图转为结构化文本推理,作为中间表示增强空间推理
  • 在多个基准上显著提升深度信息利用率和空间理解性能
  • 适合需要精细空间感知的多模态应用,如机器人导航、3D场景理解

尽管视觉语言模型(VLMs)在多模态任务中取得了显著进展,但其依赖RGB输入限制了精确的空间理解。现有融合空间线索(如点云或深度图)的方法要么需要特殊传感器,要么无法有效利用深度信息进行高阶推理。为此,我们提出一种名为SSR的新框架,将原始深度数据转化为结构化、可解释的文本推理过程。这些文本推理作为有意义的中间表示,显著增强了空间推理能力。同时,我们采用知识蒸馏将生成的推理压缩为紧凑的潜在嵌入,实现资源高效且无需重新训练即可接入现有VLMs。为支持全面评估,我们构建了名为SSR-CoT的百万级视觉语言推理数据集,包含中间空间推理标注,并提出了SSRBench多任务基准。在多个基准上的大量实验表明,SSR显著提升了深度信息的利用效率,增强了空间推理能力,推动VLMs向更类人的多模态理解迈进。

原文摘要 · Abstract (English)

Despite impressive advancements in Visual-Language Models (VLMs) for multi-modal tasks, their reliance on RGB inputs limits precise spatial understanding. Existing methods for integrating spatial cues, such as point clouds or depth, either require specialized sensors or fail to effectively exploit depth information for higher-order reasoning. To this end, we propose a novel Spatial Sense and Reasoning method, dubbed SSR, a novel framework that transforms raw depth data into structured, interpretable textual rationales. These textual rationales serve as meaningful intermediate representations to significantly enhance spatial reasoning capabilities. Additionally, we leverage knowledge distillation to compress the generated rationales into compact latent embeddings, which facilitate resource-efficient and plug-and-play integration into existing VLMs without retraining. To enable comprehensive evaluation, we introduce a new dataset named SSR-CoT, a million-scale visual-language reasoning dataset enriched with intermediate spatial reasoning annotations, and present SSRBench, a comprehensive multi-task benchmark. Extensive experiments on multiple benchmarks demonstrate SSR substantially improves depth utilization and enhances spatial reasoning, thereby advancing VLMs toward more human-like multi-modal understanding. Project page: https://yliu-cs.github.io/SSR.

空间推理视觉语言模型深度感知知识蒸馏

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。