无需训练数据,一张图就能推算任意镜头的精确深度。
Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Camera
- 用等距矩形投影统一不同视角图像,实现跨镜头处理
- 在鱼眼和360度相机上比之前方法精度提升50%以上
- 适合需要快速适配新摄像头的自动驾驶与机器人场景
尽管近期深度基础模型展现出强大的零样本泛化能力,但在不同相机类型(尤其是视场角大如鱼眼和360度相机)之间实现精确度量深度仍具挑战。本文提出深度任意相机(DAC),一种零样本度量深度估计框架,将原本基于透视图像训练的模型扩展至多种视场角相机。该框架仅使用透视图像进行训练,即可无缝推广至鱼眼和360度相机,无需额外训练数据。DAC采用等距矩形投影(ERP)作为统一图像表示,核心组件包括:感知俯仰的图像到ERP转换及在线增强,用于模拟未畸变输入下的畸变ERP块;视场角对齐操作,实现跨广范围视场角的有效训练;多分辨率数据增强,缓解训练与测试间分辨率差异。在多个鱼眼和360度数据集上,DAC的δ₁精度相比先前度量深度基础模型最高提升50%,验证了其在不同相机类型间的鲁棒泛化能力。
原文摘要 · Abstract (English)
While recent depth foundation models exhibit strong zero-shot generalization, achieving accurate metric depth across diverse camera types-particularly those with large fields of view (FoV) such as fisheye and 360-degree cameras-remains a significant challenge. This paper presents Depth Any Camera (DAC), a powerful zero-shot metric depth estimation framework that extends a perspective-trained model to effectively handle cameras with varying FoVs. The framework is designed to ensure that all existing 3D data can be leveraged, regardless of the specific camera types used in new applications. Remarkably, DAC is trained exclusively on perspective images but generalizes seamlessly to fisheye and 360-degree cameras without the need for specialized training data. DAC employs Equi-Rectangular Projection (ERP) as a unified image representation, enabling consistent processing of images with diverse FoVs. Its core components include pitch-aware Image-to-ERP conversion with efficient online augmentation to simulate distorted ERP patches from undistorted inputs, FoV alignment operations to enable effective training across a wide range of FoVs, and multi-resolution data augmentation to further address resolution disparities between training and testing. DAC achieves state-of-the-art zero-shot metric depth estimation, improving $δ_1$ accuracy by up to 50% on multiple fisheye and 360-degree datasets compared to prior metric depth foundation models, demonstrating robust generalization across camera types.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。