让视觉语言模型学会3D空间理解，仅用2D视频就能精准定位与推理。
3D-Aware VLMs with Implicit and Explicit Geometries
- 引入隐式与显式几何令牌，从2D视频中提取3D空间先验。
- 在多个3D任务上表现优于现有方法，尤其在细粒度空间推理上提升显著。
- 无需额外3D数据，适合希望增强模型空间感知的开发者使用。
尽管进展迅速，大多数基于2D视觉输入的视觉语言模型（VLMs）在需要精细空间理解与推理的3D任务中仍表现不佳。为此，我们提出VLM-IE3D，一种统一框架，通过从RGB视频中学习隐式与显式3D几何信息，增强VLM的3D空间感知能力。该框架引入隐式几何令牌（IGTs），捕捉输入视频中的高层次几何先验；同时结合显式几何令牌（EGTs），编码由重建3D属性得到的详细几何结构。此外，设计了3D感知适配器，有效融合两类几何表示与2D视觉线索。此仅依赖RGB视频的设计，为细粒度空间理解注入强3D归纳偏置，无需任何额外3D输入。大量实验表明，VLM-IE3D在3D视频检测、3D视觉定位、3D密集描述和空间推理等任务上均持续取得优异表现。代码与模型已公开于https://github.com/Vegetebird/VLM-IE3D。
原文摘要 · Abstract (English)
Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning. To bridge this gap, we present VLM-IE3D, a unified framework that enhances the 3D spatial awareness of VLMs by equipping them with both implicit and explicit 3D geometries learned from RGB videos. Our VLM-IE3D introduces Implicit Geometry Tokens (IGTs) that capture high-level geometric priors from input videos, as well as complementary Explicit Geometry Tokens (EGTs) that encode detailed geometric structures from reconstructed 3D attributes. On top of that, VLM-IE3D comes with a 3D-aware adapter that effectively fuses the two types of geometric representations with 2D visual cues. This RGB-only design injects strong 3D inductive biases for fine-grained spatial understanding and reasoning without requiring any additional 3D inputs. Extensive experiments show that VLM-IE3D achieves superior performance consistently across various 3D tasks including 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning. Code and models are available at https://github.com/Vegetebird/VLM-IE3D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。