用视觉基础模型增强自监督立体匹配,提升真实场景下的精度
SMFormer: Empowering Self-supervised Stereo Matching via Foundation Models and Data Augmentation
- 引入视觉基础模型与特征金字塔网络,提升特征鲁棒性
- 设计光照变化一致性约束的数据增强机制,强化泛化能力
- 在挑战性数据集上超越部分有监督方法,适合追求高精度自监督方案的研究者
近期自监督立体匹配方法取得显著进展,但普遍依赖光度一致性假设——即不同视角对应点具有相同外观。然而真实场景中的干扰常破坏该假设,导致无效监督信号,使性能与有监督方法差距明显。为此,本文提出SMFormer,融合视觉基础模型(VFM)与数据增强的自监督框架。首先将VFM与特征金字塔网络(FPN)结合,生成对干扰鲁棒的判别性特征表示;其次设计一种有效数据增强机制,显式强制学习特征与光照变化影响下的特征保持一致,并正则化强增强样本与标准样本的视差预测输出一致性。在多个主流基准测试中,SMFormer在自监督方法中达到当前最优性能,甚至可比肩部分有监督方法。尤其在更具挑战性的Booster基准上,其表现优于某些SOTA有监督方法(如CFNet)。
原文摘要 · Abstract (English)
Recent self-supervised stereo matching methods have made significant progress. They typically rely on the photometric consistency assumption, which presumes corresponding points across views share the same appearance. However, this assumption could be compromised by real-world disturbances, resulting in invalid supervisory signals and a significant accuracy gap compared to supervised methods. To address this issue, we propose SMFormer, a framework integrating more reliable self-supervision guided by the Vision Foundation Model (VFM) and data augmentation. We first incorporate the VFM with the Feature Pyramid Network (FPN), providing a discriminative and robust feature representation against disturbance in various scenarios. We then devise an effective data augmentation mechanism that ensures robustness to various transformations. The data augmentation mechanism explicitly enforces consistency between learned features and those influenced by illumination variations. Additionally, it regularizes the output consistency between disparity predictions of strong augmented samples and those generated from standard samples. Experiments on multiple mainstream benchmarks demonstrate that our SMFormer achieves state-of-the-art (SOTA) performance among self-supervised methods and even competes on par with supervised ones. Remarkably, in the challenging Booster benchmark, SMFormer even outperforms some SOTA supervised methods, such as CFNet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。