让视觉语言模型直接学会3D理解，无需复杂设计
VLM3: Vision Language Models Are Native 3D Learners
- 仅用焦距统一、文本像素定位和数据混合即可实现3D学习
- 深度估计准确率提升至0.9，媲美专业3D模型
- 适合希望简化3D建模流程的研究者与开发者
视觉语言模型（VLM）通过提示词可统一处理多种视觉任务，在语义理解上表现优异。但3D理解仍依赖结构复杂的专用视觉模型。本工作提出，VLM天生具备3D学习能力。大规模实验表明，仅需焦距统一、基于文本的像素定位以及数据混合与扩展，即可有效实现3D学习。模型架构调整、大模型规模、复杂数据增强及回归损失等传统3D模型核心要素并非必需。据此提出VLM3，一种设计极简的可扩展方法，使标准VLM掌握多样3D任务。VLM3将深度估计准确率从0.84大幅提升至0.9，并实现像素对应、相机位姿估计与对象级3D理解，性能媲美专家模型，同时保持标准架构与文本训练范式。我们相信VLM3开启了一种简单且可扩展的3D学习新范式。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) enable a unified model to solve various vision tasks through prompting. They have shown promising performance in semantic understanding. However, 3D understanding still largely relies on expert vision models with complex task-specific designs. The key argument this work wants to make is that VLMs are native 3D learners. Our in-depth large scale study shows that 1) focal length unification, 2) text-based pixel reference and 3) data mixture and scaling, are all you need for effective 3D learning. Model architecture changes, large models, heavy data augmentations, and complex losses including the regression formulation, many of which form the foundation of expert vision models, are actually not necessary conditions. As a result, we propose VLM3, a scalable method with the simplest design that enables standard VLMs to master diverse 3D tasks. VLM3 not only advances the VLM depth estimation accuracy by a large margin (0.84 -> 0.9), but also enables diverse 3D tasks such as pixel correspondence, camera pose estimation and object-level 3D understanding, matching expert vision model accuracy while maintaining standard architectures and text-based training. We believe VLM3 opens up a new paradigm for simple and scalable 3D learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。