让视觉语言模型学会理解3D空间结构，无需改架构也不用标注数据。
3D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation
- 从3D基础模型中提取几何线索，蒸馏注入预训练视觉语言模型。
- 在多个3D推理任务上表现优于现有方法，计算开销更低。
- 适合需要3D空间理解的多模态应用，如机器人导航与场景理解。
视觉语言模型（VLM）在多种视觉与语言任务中表现出色，但在理解三维空间结构方面仍存在根本性局限。本文提出几何蒸馏（Geometric Distillation），一种轻量级、无需标注的微调框架，可在不修改模型架构的前提下，将人类启发的几何线索注入预训练的VLM中。通过蒸馏来自现成3D基础模型（如MASt3R、VGGT）的稀疏对应关系、相对深度关系和密集代价体，该方法使模型表示具备几何感知能力，同时保持对自然图像-文本输入的兼容性。在多个3D视觉-语言推理与3D感知基准上的广泛评估表明，该方法持续优于先前方法，在提升3D空间推理能力的同时，显著降低计算成本。本工作展示了一条可扩展且高效的路径，实现2D训练的VLM与3D理解的融合，为具有空间语境的多模态任务开辟更广泛应用前景。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have shown remarkable performance on diverse visual and linguistic tasks, yet they remain fundamentally limited in their understanding of 3D spatial structures. We propose Geometric Distillation, a lightweight, annotation-free fine-tuning framework that injects human-inspired geometric cues into pretrained VLMs without modifying their architecture. By distilling (1) sparse correspondences, (2) relative depth relations, and (3) dense cost volumes from off-the-shelf 3D foundation models (e.g., MASt3R, VGGT), our method shapes representations to be geometry-aware while remaining compatible with natural image-text inputs. Through extensive evaluations on 3D vision-language reasoning and 3D perception benchmarks, our method consistently outperforms prior approaches, achieving improved 3D spatial reasoning with significantly lower computational cost. Our work demonstrates a scalable and efficient path to bridge 2D-trained VLMs with 3D understanding, opening up wider use in spatially grounded multimodal tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。