用超分与语义模型增强单目深度估计的数据质量,无需额外训练数据。
EDADepth: Enhanced Data Augmentation for Monocular Depth Estimation
- 引入超分辨率模型与语义分割模型提升输入图像和文本嵌入质量。
- 在NYUv2和KITTI上达成最优的delta3指标,其他指标接近顶尖水平。
- 适合关注扩散模型在视觉任务中数据增强的开发者和研究者。
由于具备文本到图像生成能力,扩散模型近年来被广泛应用于深度估计等视觉感知任务。然而,高质量数据集的缺乏使得扩散模型难以提取细粒度语义上下文,而语义信息不足进一步影响文本嵌入的有效性。本文提出EDADepth,一种无需额外训练数据的单目深度估计增强数据增强方法。我们采用Swin2SR超分辨率模型提升输入图像质量,使用BEiT预训练语义分割模型优化文本嵌入提取,并通过BLIP-2分词器生成文本标记。该方法首次将Swin2SR、BEiT模型和BLIP-2分词器引入基于扩散模型的单目深度估计流程。实验表明,该模型在NYUv2和KITTI数据集上均达到delta3指标的当前最优(SOTA)表现,且在RMSE和REL指标上与主流模型相当。此外,深度图可视化效果也优于现有扩散模型方法。
原文摘要 · Abstract (English)
Due to their text-to-image synthesis feature, diffusion models have recently seen a rise in visual perception tasks, such as depth estimation. The lack of good-quality datasets makes the extraction of a fine-grain semantic context challenging for the diffusion models. The semantic context with fewer details further worsens the process of creating effective text embeddings that will be used as input for diffusion models. In this paper, we propose a novel EDADepth, an enhanced data augmentation method to estimate monocular depth without using additional training data. We use Swin2SR, a super-resolution model, to enhance the quality of input images. We employ the BEiT pre-trained semantic segmentation model for better extraction of text embeddings. We use BLIP-2 tokenizer to generate tokens from these text embeddings. The novelty of our approach is the introduction of Swin2SR, the BEiT model, and the BLIP-2 tokenizer in the diffusion-based pipeline for the monocular depth estimation. Our model achieves state-of-the-art results (SOTA) on the delta3 metric on NYUv2 and KITTI datasets. It also achieves results comparable to those of the SOTA models in the RMSE and REL metrics. Finally, we also show improvements in the visualization of the estimated depth compared to the SOTA diffusion-based monocular depth estimation models. Code: https://github.com/edadepthmde/EDADepth_ICMLA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。