arXiv:2506.13387cs.CV2025-06被引 2

用语言和图像指导,把模糊深度转成精确距离。

TR2M: Transferring Monocular Relative Depth to Metric Depth with Language Descriptions and Dual-Level Scale-Oriented Contrast

  • 融合图文信息,像素级生成缩放图修正深度尺度
  • 仅用少量参数,在多个数据集上表现优异,零样本泛化强
  • 适合需要跨域精确深度的应用,如机器人导航

本文提出TR2M框架,实现相对深度向度量深度的通用转换。现有单目深度估计方法分为度量深度(MMDE)与相对深度(MRDE),前者精度高但领域受限,后者泛化好但尺度不确定。为解决尺度不确定性问题,TR2M引入语言描述与图像双模态输入,通过交叉模态注意力融合特征,生成两个缩放图,在像素层面将相对深度转为度量深度。设计伪度量深度构建与筛选策略,提供更全面监督;引入面向尺度的对比学习,利用深度分布引导模型学习内在尺度知识。TR2M仅需少量可训练参数,在多领域数据集上训练后,不仅在已见数据集表现优异,更在五个未见数据集上展现卓越零样本能力。证明了语言辅助下像素级深度尺度转换的巨大潜力。(代码地址:https://github.com/BeileiCui/TR2M)

原文摘要 · Abstract (English)

This work presents a generalizable framework to transfer relative depth to metric depth. Current monocular depth estimation methods are mainly divided into metric depth estimation (MMDE) and relative depth estimation (MRDE). MMDEs estimate depth in metric scale but are often limited to a specific domain. MRDEs generalize well across different domains, but with uncertain scales which hinders downstream applications. To this end, we aim to build up a framework to solve scale uncertainty and transfer relative depth to metric depth. Previous methods used language as input and estimated two factors for conducting rescaling. Our approach, TR2M, utilizes both text description and image as inputs and estimates two rescale maps to transfer relative depth to metric depth at pixel level. Features from two modalities are fused with a cross-modality attention module to better capture scale information. A strategy is designed to construct and filter confident pseudo metric depth for more comprehensive supervision. We also develop scale-oriented contrastive learning to utilize depth distribution as guidance to enforce the model learning about intrinsic knowledge aligning with the scale distribution. TR2M only exploits a small number of trainable parameters to train on datasets in various domains and experiments not only demonstrate TR2M's great performance in seen datasets but also reveal superior zero-shot capabilities on five unseen datasets. We show the huge potential in pixel-wise transferring relative depth to metric depth with language assistance. (Code is available at: https://github.com/BeileiCui/TR2M)

深度估计图文对齐零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。