arXiv:2409.12202cs.CV2024-09被引 4

仅用两张图片实现精准3D运动估计,突破现有方法稳定性瓶颈。

ScaleFlow++: Robust and Accurate Estimation of 3D Motion from Video

  • 通过跨尺度匹配提取深层运动线索,统一建模光流与深度运动。
  • 在KITTI上将场景流误差降至5.79,深度运动估计超越RGBD方法。
  • 零样本泛化能力强,适用于刚性与非刚性场景,适合自动驾驶等应用。

三维运动感知是自动驾驶、机器人和运动预测等领域核心技术。本文提出一种名为ScaleFlow++的3D运动感知方法,仅需一对RGB图像即可鲁棒地估计光流与运动深度(MID)。现有方法直接从双帧或光流回归MID,导致结果不准确且不稳定。本文关键洞察为跨尺度匹配,通过在不同尺度下匹配图像对中的物体来提取深层运动线索。与以往方法不同,ScaleFlow++将光流与MID估计整合至统一架构中,基于特征匹配端到端联合估计。此外,还设计了全局初始化网络、全局迭代优化器及混合训练流程,以融合全局运动信息、减少迭代次数并防止训练过拟合。在KITTI数据集上,ScaleFlow++实现了最佳单目场景流估计性能,将SF-all从6.21降至5.79;MID评估结果甚至超越基于RGBD的方法。同时,在刚性与非刚性场景下均展现出卓越的零样本泛化能力。代码已开源:https://github.com/HanLingsgjk/CSCV。

原文摘要 · Abstract (English)

Perceiving and understanding 3D motion is a core technology in fields such as autonomous driving, robots, and motion prediction. This paper proposes a 3D motion perception method called ScaleFlow++ that is easy to generalize. With just a pair of RGB images, ScaleFlow++ can robustly estimate optical flow and motion-in-depth (MID). Most existing methods directly regress MID from two RGB frames or optical flow, resulting in inaccurate and unstable results. Our key insight is cross-scale matching, which extracts deep motion clues by matching objects in pairs of images at different scales. Unlike previous methods, ScaleFlow++ integrates optical flow and MID estimation into a unified architecture, estimating optical flow and MID end-to-end based on feature matching. Moreover, we also proposed modules such as global initialization network, global iterative optimizer, and hybrid training pipeline to integrate global motion information, reduce the number of iterations, and prevent overfitting during training. On KITTI, ScaleFlow++ achieved the best monocular scene flow estimation performance, reducing SF-all from 6.21 to 5.79. The evaluation of MID even surpasses RGBD-based methods. In addition, ScaleFlow++ has achieved stunning zero-shot generalization performance in both rigid and nonrigid scenes. Code is available at \url{https://github.com/HanLingsgjk/CSCV}.

3D运动估计光流单目感知跨尺度匹配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。