让3D感知在视觉模型中自然涌现,无需额外计算开销。
3D-IDE: 3D Implicit Depth Emergent

- 通过几何自监督构建信息瓶颈,让3D结构从视觉特征中自发生成。
- 推理延迟降低55%,且在多个3D任务上超越现有最佳方法。
- 适合追求高效、无依赖3D理解的视觉语言模型研究者。
将3D信息融入多模态大语言模型(MLLMs)近期在室内场景理解中展现出显著优势。然而,现有方法包括使用显式真实3D位置编码或外接3D基础模型来隐式建模几何,均面临2D-3D表示融合的权衡问题,导致部署效果不佳。为此,我们提出3D-Implicit Depth Emergence(3D-IDE),将3D感知重新定义为由几何自监督催生的涌现特性,而非显式编码。核心思想是隐式几何涌现原则:通过细粒度几何验证器和全局表示约束等机制施加特权几何监督,构建信息瓶颈,迫使模型最大化视觉特征与3D结构间的互信息,使3D意识在统一视觉表征中自然浮现。相比现有方法,本方法实现3D感知的隐式涌现,解耦密集区域特征,且推理时完全消除深度与位姿依赖,零延迟开销。这一从外部拼接到内在涌现的根本范式转变,推动了视觉语言模型中3D知识整合的新思路。大量实验表明,该方法在多个3D场景理解基准上超越当前最优水平,同时实现55%的推理延迟下降,且在多样下游任务中保持强性能,验证了精心设计辅助目标对无依赖3D理解的有效性。源代码见github.com/ChushanZhang/3D-IDE。
原文摘要 · Abstract (English)
Leveraging 3D information within Multimodal Large Language Models (MLLMs) has recently shown significant advantages for indoor scene understanding. However, existing methods, including those using explicit ground-truth 3D positional encoding and those grafting external 3D foundation models for implicit geometry, struggle with the trade-off in 2D-3D representation fusion, leading to suboptimal deployment. To this end, we propose 3D-Implicit Depth Emergence, a method that reframes 3D perception as an emergent property derived from geometric self-supervision rather than explicit encoding. Our core insight is the Implicit Geometric Emergence Principle: by strategically leveraging privileged geometric supervision through mechanisms like a fine-grained geometry validator and global representation constraints, we construct an information bottleneck. This bottleneck forces the model to maximize the mutual information between visual features and 3D structures, allowing 3D awareness to emerge naturally within a unified visual representation. Unlike existing approaches, our method enables 3D perception to emerge implicitly, disentangling features in dense regions and, crucially, eliminating depth and pose dependencies during inference with zero latency overhead. This paradigm shift from external grafting to implicit emergence represents a fundamental rethinking of 3D knowledge integration in visual-language models. Extensive experiments demonstrate that our method surpasses SOTA on multiple 3D scene understanding benchmarks. Our approach achieves a 55% reduction in inference latency while maintaining strong performance across diverse downstream tasks, underscoring the effectiveness of meticulously designed auxiliary objectives for dependency-free 3D understanding. Source code can be found at github.com/ChushanZhang/3D-IDE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。