用掩码图像建模提升双目匹配,让Transformer更懂局部结构。
MaDis-Stereo: Enhanced Stereo Matching via Distilled Masked Image Modeling
- 用掩码图像重建+深度预测联合训练,增强Transformer的局部先验
- 在ETH3D和KITTI 2015上达到当前最优性能
- 通过教师-学生蒸馏稳定训练,适合追求高精度双目匹配的研究者
在双目匹配中,传统以CNN为主。尽管近期已有基于Transformer的模型研究,但因数据稀缺问题,其性能仍落后于CNN模型。本文提出基于掩码图像建模的蒸馏双目匹配方法MaDis-Stereo,通过在训练中引入掩码图像建模,增强Transformer模型的局部性先验。给定随机遮盖的双目图像作为输入,模型同时尝试完成图像重建与深度预测任务。该策略虽有助于缓解数据稀缺问题,但重建被遮盖区域并进行双目匹配带来的双重挑战,尤其影响训练稳定性。为此,我们引入一个通过指数移动平均(EMA)更新的辅助网络(教师),与原始模型(学生)协同训练,利用教师预测提供伪监督信号,实现知识有效蒸馏。所提方法在ETH3D和KITTI 2015等多个双目匹配基准上取得当前最优结果。此外,为验证模型对局部性先验的有效利用,我们提供了注意力距离测量结果。
原文摘要 · Abstract (English)
In stereo matching, CNNs have traditionally served as the predominant architectures. Although Transformer-based stereo models have been studied recently, their performance still lags behind CNN-based stereo models due to the inherent data scarcity issue in the stereo matching task. In this paper, we propose Masked Image Modeling Distilled Stereo matching model, termed MaDis-Stereo, that enhances locality inductive bias by leveraging Masked Image Modeling (MIM) in training Transformer-based stereo model. Given randomly masked stereo images as inputs, our method attempts to conduct both image reconstruction and depth prediction tasks. While this strategy is beneficial to resolving the data scarcity issue, the dual challenge of reconstructing masked tokens and subsequently performing stereo matching poses significant challenges, particularly in terms of training stability. To address this, we propose to use an auxiliary network (teacher), updated via Exponential Moving Average (EMA), along with the original stereo model (student), where teacher predictions serve as pseudo supervisory signals to effectively distill knowledge into the student model. State-of-the-arts performance is achieved with the proposed method on several stereo matching such as ETH3D and KITTI 2015. Additionally, to demonstrate that our model effectively leverages locality inductive bias, we provide the attention distance measurement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。