大模型能用语言理解实现单目深度估计，少样本下效果好且省资源。
Large Language Models Can Understanding Depth from Monocular Images
- 用语言理解引导视觉特征，通过跨模态重编程对齐图文表示
- 零/少样本下在多个真实数据集上表现优于传统方法
- 自动生成提示词，降低人工干预，适合轻量化部署场景
单目深度估计是计算机视觉中的关键任务。本文表明，大型语言模型（LLMs）可在极少监督条件下有效理解深度信息，具备高效资源利用与一致神经网络架构。我们提出 LLM-MDE，一种通过语言理解解码深度的多模态框架。具体而言，该框架采用两种策略增强预训练大模型的深度估计能力：跨模态重编程与自适应提示生成模块。前者将视觉表征与文本原型对齐，后者根据单目图像自动生成提示。在真实世界单目深度估计数据集上的全面实验验证了 LLM-MDE 的有效性与优越性，尤其在少样本/零样本任务中表现突出，同时显著降低资源消耗。源代码已公开。
原文摘要 · Abstract (English)
Monocular depth estimation is a critical function in computer vision applications. This paper shows that large language models (LLMs) can effectively interpret depth with minimal supervision, using efficient resource utilization and a consistent neural network architecture. We introduce LLM-MDE, a multimodal framework that deciphers depth through language comprehension. Specifically, LLM-MDE employs two main strategies to enhance the pretrained LLM's capability for depth estimation: cross-modal reprogramming and an adaptive prompt estimation module. These strategies align vision representations with text prototypes and automatically generate prompts based on monocular images, respectively. Comprehensive experiments on real-world MDE datasets confirm the effectiveness and superiority of LLM-MDE, which excels in few-/zero-shot tasks while minimizing resource use. The source code is available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。