用单目深度模型提升稀疏深度图补全,无需真实深度数据。
StarryGazer: Leveraging Monocular Depth Estimation Models for Domain-Agnostic Single Depth Image Completion
- 用预训练单目深度模型生成相对深度图,构建合成伪真值对。
- 通过随机缩放与分割生成训练数据,提升补全精度与鲁棒性。
- 不依赖真实深度,适用于多种场景,适合无标注数据环境。
深度补全任务是从单张稀疏深度图和彩色图像中预测稠密深度图。无监督方法常需辅助数据估算深度,脱离真实场景。单目深度估计(MDE)模型可从单图生成合理相对深度图,但现有工作未有效融合稀疏深度与MDE输出;简单仿射变换会导致误差,因MDE在物体间深度差估计上不准。本文提出StarryGazer,一种无需真实深度的领域无关框架,利用大尺寸MDE模型从单张稀疏深度图与RGB图像生成稠密深度图。首先使用预训练MDE模型生成相对深度图,经分割与随机缩放形成合成伪真值对及对应稀疏深度。再以该合成数据训练精炼网络,结合相对深度图与RGB图像提升准确性与鲁棒性。在多个数据集上,StarryGazer优于现有无监督方法及直接转换的MDE结果,证明其有效利用MDE能力并借助稀疏深度修正误差。
原文摘要 · Abstract (English)
The problem of depth completion involves predicting a dense depth image from a single sparse depth map and an RGB image. Unsupervised depth completion methods have been proposed for various datasets where ground truth depth data is unavailable and supervised methods cannot be applied. However, these models require auxiliary data to estimate depth values, which is far from real scenarios. Monocular depth estimation (MDE) models can produce a plausible relative depth map from a single image, but there is no work to properly combine the sparse depth map with MDE for depth completion; a simple affine transformation to the depth map will yield a high error since MDE are inaccurate at estimating depth difference between objects. We introduce StarryGazer, a domain-agnostic framework that predicts dense depth images from a single sparse depth image and an RGB image without relying on ground-truth depth by leveraging the power of large MDE models. First, we employ a pre-trained MDE model to produce relative depth images. These images are segmented and randomly rescaled to form synthetic pairs for dense pseudo-ground truth and corresponding sparse depths. A refinement network is trained with the synthetic pairs, incorporating the relative depth maps and RGB images to improve the model's accuracy and robustness. StarryGazer shows superior results over existing unsupervised methods and transformed MDE results on various datasets, demonstrating that our framework exploits the power of MDE models while appropriately fixing errors using sparse depth information.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。