从视频中自监督学习物体刚性嵌入，实现无标记的动态场景重建。
SIRE: SE(3) Intrinsic Rigidity Embeddings
- 通过4D重建损失端到端训练图像编码器，估计场景刚性与几何。
- 在单视频上优化即可捕捉特定场景结构，数据效率高。
- 适用于物体分割、刚体运动估计等任务，仅需少量监督。
运动是场景感知和理解的重要线索，能分离独立运动的表面并把物理世界组织为独立实体。我们提出SIRE，一种从随意视频中自监督发现物体运动并重建动态场景的方法，通过学习场景的内在刚性嵌入实现。该方法训练图像编码器以估计场景刚性和几何，利用简单的4D重建损失进行监督：一个最小二乘求解器将估计的几何与刚性信息用于将2D点轨迹提升至SE(3)空间，再投影回2D，与原始2D轨迹对比以获得监督信号。关键在于，该框架完全可微且端到端，既可在视频数据集上训练以学习通用图像先验，也可在单个视频上优化以捕捉特定场景结构——体现强数据效率。我们在多个设置下验证了刚性嵌入与几何的有效性，包括下游物体分割、SE(3)刚体运动估计以及自监督深度估计。结果表明，SIRE能从视频数据中学习强大的几何与运动刚性先验，仅需极少监督。
原文摘要 · Abstract (English)
Motion serves as a powerful cue for scene perception and understanding by separating independently moving surfaces and organizing the physical world into distinct entities. We introduce SIRE, a self-supervised method for motion discovery of objects and dynamic scene reconstruction from casual scenes by learning intrinsic rigidity embeddings from videos. Our method trains an image encoder to estimate scene rigidity and geometry, supervised by a simple 4D reconstruction loss: a least-squares solver uses the estimated geometry and rigidity to lift 2D point track trajectories into SE(3) tracks, which are simply re-projected back to 2D and compared against the original 2D trajectories for supervision. Crucially, our framework is fully end-to-end differentiable and can be optimized either on video datasets to learn generalizable image priors, or even on a single video to capture scene-specific structure - highlighting strong data efficiency. We demonstrate the effectiveness of our rigidity embeddings and geometry across multiple settings, including downstream object segmentation, SE(3) rigid motion estimation, and self-supervised depth estimation. Our findings suggest that SIRE can learn strong geometry and motion rigidity priors from video data, with minimal supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。