arXiv:2503.15022cs.CV2025-03CVPR被引 1

用2D运动信息提升3D多物体发现,跨模态训练更准更快。

xMOD: Cross-Modal Distillation for 2D/3D Multi-Object Discovery from 2D motion

  • 用2D运动线索指导3D物体发现,结合场景补全实现稀疏输入下的密集定位
  • 在真实数据集上比现有2D方法提升8.7%~15.1% F1@50得分
  • 支持纯图像或点云输入,双模态时用融合策略进一步提效

物体发现旨在无标注情况下定位物体,已在2D图像分析中受到广泛关注。然而在3D数据中仍研究不足,现有方法仅依赖3D运动,面临诸多挑战。本文提出新框架xMOD,利用基于2D运动的2D物体发现进展,借助2D运动更灵活、泛化性更强的优势,弥合2D与3D模态差距。主要贡献有二:(i) 提出DIOD-3D,首个基于2D运动的3D多物体发现基线,引入场景补全作为辅助任务,实现从稀疏输入中密集定位物体;(ii) 设计xMOD跨模态训练框架,整合2D与3D数据,始终使用2D运动线索。采用跨模态师生训练缓解确认偏见。推理时支持仅RGB或仅点云输入;此外提出专为该流程设计的后融合技术,双模态可用时性能更优。在合成数据集TRIP-PD及真实世界数据集KITTI、Waymo上广泛评估,相比2D物体发现最先进方法,所有数据集上F1@50得分提升8.7至15.1。代码已开源。

原文摘要 · Abstract (English)

Object discovery, which refers to the task of localizing objects without human annotations, has gained significant attention in 2D image analysis. However, despite this growing interest, it remains under-explored in 3D data, where approaches rely exclusively on 3D motion, despite its several challenges. In this paper, we present a novel framework that leverages advances in 2D object discovery which are based on 2D motion to exploit the advantages of such motion cues being more flexible and generalizable and to bridge the gap between 2D and 3D modalities. Our primary contributions are twofold: (i) we introduce DIOD-3D, the first baseline for multi-object discovery in 3D data using 2D motion, incorporating scene completion as an auxiliary task to enable dense object localization from sparse input data; (ii) we develop xMOD, a cross-modal training framework that integrates 2D and 3D data while always using 2D motion cues. xMOD employs a teacher-student training paradigm across the two modalities to mitigate confirmation bias by leveraging the domain gap. During inference, the model supports both RGB-only and point cloud-only inputs. Additionally, we propose a late-fusion technique tailored to our pipeline that further enhances performance when both modalities are available at inference. We evaluate our approach extensively on synthetic (TRIP-PD) and challenging real-world datasets (KITTI and Waymo). Notably, our approach yields a substantial performance improvement compared with the 2D object discovery state-of-the-art on all datasets with gains ranging from +8.7 to +15.1 in F1@50 score. The code is available at https://github.com/CEA-LIST/xMOD

3D物体发现跨模态学习运动线索无监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。