用二值局部排序图融合单目先验,提升立体匹配在遮挡等难题上的泛化能力。
Diving into the Fusion of Monocular Priors for Generalized Stereo Matching
- 设计二值局部排序图统一相对与绝对深度表示,解决单目先验与视差对齐问题。
- 通过重加权初始视差更新,缓解迭代过程中的过自信和局部最优问题。
- 将单目深度直接融合建模为像素级线性配准,全局自适应对齐提升精度。
立体匹配在遮挡、非朗伯表面等病态区域难以处理。虽已有研究利用单目先验改善此类问题,但小规模立体数据集训练的单目先验存在偏差,限制了泛化能力。近期,基于视觉基础模型(VFM)的无偏单目先验提升了病态区域的泛化性能。本文深入分析其融合过程,发现三大瓶颈:仿射不变的相对单目深度与视差的绝对深度存在错位;在迭代更新结构中,单目特征导致视差更新过自信,陷入局部最优;直接融合单目深度可缓解局部最优,但前几轮计算的噪声视差会误导融合。为此,本文提出二值局部排序图,将深度图转换为二值相对格式,统一相对与绝对深度表达;同时利用该排序图重加权初始视差更新,缓解局部最优与噪声问题。此外,将最终的单目深度与视差直接融合建模为注册问题,采用像素级线性回归模块实现全局自适应对齐。实验表明,在从SceneFlow到Middlebury和Booster数据集的跨域测试中,性能显著提升,且计算效率几乎无损。
原文摘要 · Abstract (English)
The matching formulation makes it naturally hard for the stereo matching to handle ill-posed regions like occlusions and non-Lambertian surfaces. Fusing monocular priors has been proven helpful for ill-posed matching, but the biased monocular prior learned from small stereo datasets constrains the generalization. Recently, stereo matching has progressed by leveraging the unbiased monocular prior from the vision foundation model (VFM) to improve the generalization in ill-posed regions. We dive into the fusion process and observe three main problems limiting the fusion of the VFM monocular prior. The first problem is the misalignment between affine-invariant relative monocular depth and absolute depth of disparity. Besides, when we use the monocular feature in an iterative update structure, the over-confidence in the disparity update leads to local optima results. A direct fusion of a monocular depth map could alleviate the local optima problem, but noisy disparity results computed at the first several iterations will misguide the fusion. In this paper, we propose a binary local ordering map to guide the fusion, which converts the depth map into a binary relative format, unifying the relative and absolute depth representation. The computed local ordering map is also used to re-weight the initial disparity update, resolving the local optima and noisy problem. In addition, we formulate the final direct fusion of monocular depth to the disparity as a registration problem, where a pixel-wise linear regression module can globally and adaptively align them. Our method fully exploits the monocular prior to support stereo matching results effectively and efficiently. We significantly improve the performance from the experiments when generalizing from SceneFlow to Middlebury and Booster datasets while barely reducing the efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。