用新型Mamba模块实现光学流与立体匹配的实时高精度联合估计
DenVisCoM: Dense Vision Correspondence Mamba for Efficient and Real-time Optical Flow and Stereo Estimation
- 基于Mamba与Transformer混合架构，提升多视角几何任务效率
- 在多个数据集上实现高精度光学流与深度图实时计算（>30FPS）
- 适合自动驾驶、机器人等需实时3D感知的场景
本文提出一种新型Mamba块DenVisCoM及专为光学流与视差估计设计的混合架构，针对多视角几何与运动估计任务的本质关联性，构建统一框架实现联合求解。该架构结合DenVisCoM与基于Transformer的注意力模块，在保证高精度的同时，显著优化实时推理速度与内存占用。在大量数据集上的实验表明，模型可在超过30帧/秒下完成精确的光学流与视差估计。所有模型与代码已开源于https://github.com/vimstereo/DenVisCoM。
原文摘要 · Abstract (English)
In this work, we propose a novel Mamba block DenVisCoM, as well as a novel hybrid architecture specifically tailored for accurate and real-time estimation of optical flow and disparity estimation. Given that such multi-view geometry and motion tasks are fundamentally related, we propose a unified architecture to tackle them jointly. Specifically, the proposed hybrid architecture is based on DenVisCoM and a Transformer-based attention block that efficiently addresses real-time inference, memory footprint, and accuracy at the same time for joint estimation of motion and 3D dense perception tasks. We extensively analyze the benchmark trade-off of accuracy and real-time processing on a large number of datasets. Our experimental results and related analysis suggest that our proposed model can accurately estimate optical flow and disparity estimation in real time. All models and associated code are available at https://github.com/vimstereo/DenVisCoM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。