arXiv:2509.25413cs.CV2025-09被引 27

用文本提示让视觉语言模型实现高精度深度估计,无需修改结构或损失函数。

DepthLM: Metric Depth From Vision Language Models

论文配图:DepthLM: Metric Depth From Vision Language Models
图 1 · 摘自论文原文
  • 通过文本监督微调与稀疏标签,无需密集预测头或复杂损失函数。
  • 小模型性能超越多数先进VLM,精度超2倍,接近纯视觉模型水平。
  • 方法简洁可扩展,单个模型可处理多种3D任务,适合通用视觉系统研究者。

视觉语言模型(VLMs)可通过文本交互灵活应对各类视觉任务。尽管在语义理解上表现优异,当前最先进的VLM(如GPT-5)仍难以从2D输入中理解3D信息。相比之下,专用视觉模型在度量深度估计这一关键3D理解任务上已达到超人水平,但需特定架构和损失函数。这促使我们思考:能否在不改变架构或损失的前提下,让VLM达到专家级精度?我们以像素级度量深度估计为代表任务,发现答案是肯定的!出人意料的是,分析表明,仅使用带稀疏标签的文本监督微调即可激活VLM强大的3D理解能力,无需密集预测头或复杂回归/正则化损失。瓶颈在于像素参照与跨数据集相机模糊性,我们通过视觉提示和内在条件增强加以解决。使用更小的模型,DepthLM的精度超过大多数先进VLM的两倍以上,首次使VLM在该任务上媲美纯视觉模型。有趣的是,训练中未显式约束,但VLM自然避免过度平滑,在边界区域飞点数量远少于纯视觉模型。DepthLM的简洁性还支持单个VLM覆盖多种3D任务。代码与模型将公开发布。

原文摘要 · Abstract (English)

Vision language models (VLMs) can flexibly address various vision tasks through text interactions. Although successful in semantic understanding, state-of-the-art VLMs including GPT-5 still struggle in understanding 3D from 2D inputs. On the other hand, expert pure vision models achieve super-human accuracy in metric depth estimation, a key 3D understanding task. However, they require task-specific architectures and losses. Such difference motivates us to ask: Can VLMs reach expert-level accuracy without architecture or loss change? We take per-pixel metric depth estimation as the representative task and show that the answer is yes! Surprisingly, comprehensive analysis shows that text-based supervised-finetuning with sparse labels is sufficient for VLMs to unlock strong 3D understanding, no dense prediction head or complex regression/regularization loss is needed. The bottleneck for VLMs lies actually in pixel reference and cross-dataset camera ambiguity, which we address through visual prompting and intrinsic-conditioned augmentation. With much smaller models, our method DepthLM surpasses the accuracy of most advanced VLMs by over 2x, making VLMs for the first time comparable with pure vision models. Interestingly, without explicit enforcement during training, VLMs trained with DepthLM naturally avoids over-smoothing, having much fewer flying points at boundary regions than pure vision models. The simplicity of DepthLM also enables a single VLM to cover various 3D tasks beyond metric depth. Our code and model will be released at the link below.

深度估计视觉语言模型3D理解文本提示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。