arXiv:2505.21635cs.CV2025-05NeurIPS

从视频运动中无监督学习物体概念,无需标注或相机参数。

Object Concepts Emerge from Motion

  • 用光流和聚类生成运动掩码,作为伪实例标签训练模型。
  • 在深度估计、3D检测等任务上优于已有自监督方法。
  • 适合做无标注视频理解的视觉基础模型研究者使用。

物体概念是人类视觉认知的基础,使我们能够感知、记忆并互动于物理世界。受发展神经科学启发——婴儿通过观察运动习得物体概念——我们提出一种生物启发的无监督框架,用于学习物体中心的视觉表示。核心思想是:运动边界可作为物体级分组的强信号,可用于从原始视频中生成伪实例监督。具体而言,利用现成的光流与聚类算法生成基于运动的实例掩码,并通过对比学习训练视觉编码器。该框架完全无需标签,且不依赖相机标定,可扩展至大规模非结构化视频数据。我们在三个下游任务上评估,涵盖低层(单目深度估计)与高层(3D物体检测和占据预测)。模型表现超越先前的监督与自监督基线,在未见场景中展现出强泛化能力。结果表明,运动诱导的物体表示为现有视觉基础模型提供了一种有前景的替代方案,捕捉了一个关键但被忽视的抽象层次:视觉实例。代码将在论文接收后公开。

原文摘要 · Abstract (English)

Object concepts play a foundational role in human visual cognition, enabling perception, memory, and interaction in the physical world. Inspired by findings in developmental neuroscience - where infants are shown to acquire object understanding through observation of motion - we propose a biologically inspired framework for learning object-centric visual representations in an unsupervised manner. Our key insight is that motion boundary serves as a strong signal for object-level grouping, which can be used to derive pseudo instance supervision from raw videos. Concretely, we generate motion-based instance masks using off-the-shelf optical flow and clustering algorithms, and use them to train visual encoders via contrastive learning. Our framework is fully label-free and does not rely on camera calibration, making it scalable to large-scale unstructured video data. We evaluate our approach on three downstream tasks spanning both low-level (monocular depth estimation) and high-level (3D object detection and occupancy prediction) vision. Our models outperform previous supervised and self-supervised baselines and demonstrate strong generalization to unseen scenes. These results suggest that motion-induced object representations offer a compelling alternative to existing vision foundation models, capturing a crucial but overlooked level of abstraction: the visual instance. The corresponding code will be released upon paper acceptance.

无监督学习物体表示视频理解运动感知

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。