arXiv:2411.09151cs.CV2024-11被引 6

用单目深度知识提升立体匹配性能,解决真实数据标签稀疏问题。

Mono2Stereo: Monocular Knowledge Transfer for Enhanced Stereo Matching

  • 通过单目图像生成合成立体图像,结合边缘感知修复填补缺失内容。
  • 提出稀疏到稠密知识蒸馏策略,使预测分布对齐稠密单目深度。
  • 预训练模型零样本泛化强,微调后在真实场景性能显著提升。

现有合成数据集与真实数据间的域差距,以及真实数据中地面真值标签的稀疏性,限制了立体匹配网络的泛化能力与性能。相比之下,单目深度估计得益于大规模深度数据集和自监督策略取得了显著进展。为弥合单目深度估计与立体匹配之间的性能差距,本文提出利用单目知识迁移增强立体匹配,即Mono2Stereo。该方法采用两阶段训练:首先在合成数据上预训练,设计数据生成流水线,从单目图像生成立体图像,利用单目深度进行图像扭曲和新视角合成,并引入提出的边缘感知(EA)修补模块填补生成图像中的缺失内容;其次在真实数据上微调,提出稀疏到稠密知识蒸馏(S2DKD)策略,促使模型预测分布与稠密单目深度对齐,缓解真实稀疏标签导致的边缘模糊问题,提升整体一致性。实验表明,预训练模型具备强大的零样本泛化能力;结合该模型与S2DKD策略进行领域特定微调,显著提升域内性能。代码即将开源。

原文摘要 · Abstract (English)

The generalization and performance of stereo matching networks are limited due to the domain gap of the existing synthetic datasets and the sparseness of GT labels in the real datasets. In contrast, monocular depth estimation has achieved significant advancements, benefiting from large-scale depth datasets and self-supervised strategies. To bridge the performance gap between monocular depth estimation and stereo matching, we propose leveraging monocular knowledge transfer to enhance stereo matching, namely Mono2Stereo. We introduce knowledge transfer with a two-stage training process, comprising synthetic data pre-training and real-world data fine-tuning. In the pre-training stage, we design a data generation pipeline that synthesizes stereo training data from monocular images. This pipeline utilizes monocular depth for warping and novel view synthesis and employs our proposed Edge-Aware (EA) inpainting module to fill in missing contents in the generated images. In the fine-tuning stage, we introduce a Sparse-to-Dense Knowledge Distillation (S2DKD) strategy encouraging the distributions of predictions to align with dense monocular depths. This strategy mitigates issues with edge blurring in sparse real-world labels and enhances overall consistency. Experimental results demonstrate that our pre-trained model exhibits strong zero-shot generalization capabilities. Furthermore, domain-specific fine-tuning using our pre-trained model and S2DKD strategy significantly increments in-domain performance. The code will be made available soon.

立体匹配单目深度知识迁移自监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。