提升雨雾黑夜下立体匹配鲁棒性,通过引入视觉先验和场景对应关系。
RoSe: Robust Self-supervised Stereo Matching under Adverse Weather Conditions
- 用视觉基础模型先验增强特征提取,对抗恶劣天气干扰
- 构建含真实天气退化的合成数据集,保持场景一致性
- 提出双阶段训练框架,显著提升夜间/雨雾中的匹配精度
近期自监督立体匹配方法进展显著,但在夜间、雨天、雾天等恶劣天气下性能急剧下降。我们识别出两个主要缺陷:一是恶劣天气引入噪声并降低可见度,使基于CNN的特征提取器难以处理反光和无纹理区域;二是这些退化区域破坏像素对应关系,导致基于光度一致性假设的监督失效。为此,我们提出将源自视觉基础模型的鲁棒先验注入CNN特征提取器,以改善恶劣天气下的特征表示。同时引入场景对应先验,构建不依赖光度一致性的鲁棒监督信号。具体地,我们生成了包含真实天气退化的合成立体数据集,其中清晰图像与恶劣天气图像保持相同的语义上下文和视差,保留场景对应特性。基于此,提出一种鲁棒自监督训练范式,包含两个关键步骤:鲁棒自监督场景对应学习与恶劣天气蒸馏。两者均旨在对齐干净与恶劣图像对的底层场景结果,从而提升模型在恶劣天气下的视差估计能力。大量实验验证了所提方法的有效性和通用性,优于现有最先进自监督方法。代码已公开于:https://github.com/cocowy1/RoSe-Robust-Self-supervised-Stereo-Matching-under-Adverse-Weather-Conditions。
原文摘要 · Abstract (English)
Recent self-supervised stereo matching methods have made significant progress, but their performance significantly degrades under adverse weather conditions such as night, rain, and fog. We identify two primary weaknesses contributing to this performance degradation. First, adverse weather introduces noise and reduces visibility, making CNN-based feature extractors struggle with degraded regions like reflective and textureless areas. Second, these degraded regions can disrupt accurate pixel correspondences, leading to ineffective supervision based on the photometric consistency assumption. To address these challenges, we propose injecting robust priors derived from the visual foundation model into the CNN-based feature extractor to improve feature representation under adverse weather conditions. We then introduce scene correspondence priors to construct robust supervisory signals rather than relying solely on the photometric consistency assumption. Specifically, we create synthetic stereo datasets with realistic weather degradations. These datasets feature clear and adverse image pairs that maintain the same semantic context and disparity, preserving the scene correspondence property. With this knowledge, we propose a robust self-supervised training paradigm, consisting of two key steps: robust self-supervised scene correspondence learning and adverse weather distillation. Both steps aim to align underlying scene results from clean and adverse image pairs, thus improving model disparity estimation under adverse weather effects. Extensive experiments demonstrate the effectiveness and versatility of our proposed solution, which outperforms existing state-of-the-art self-supervised methods. Codes are available at \textcolor{blue}{https://github.com/cocowy1/RoSe-Robust-Self-supervised-Stereo-Matching-under-Adverse-Weather-Conditions}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。