用真实视频训练无监督视频实例分割，通过关键帧蒸馏提升分割质量。
S2D: Sparse-To-Dense Keymask Distillation for Unsupervised Video Instance Segmentation
- 从单帧分割出发，利用运动先验筛选高质量关键帧作为伪标签。
- 提出稀疏到稠密蒸馏方法，结合时序丢弃损失，实现精准掩码传播。
- 完全基于真实视频训练，在多个基准上超越当前最佳表现。
近年来，无监督视频实例分割的最先进方法严重依赖从以物体为中心的图像数据集（如ImageNet）生成的合成视频数据。然而，通过人为移动和缩放图像实例掩码来合成视频，无法准确建模真实视频中的运动特性，例如视角变化、单个或多个实例的部分运动、以及相机运动。为解决此问题，我们提出一种仅在真实视频数据上训练的无监督视频实例分割模型。从单个视频帧的无监督实例分割掩码开始，这些单帧分割存在时间噪声且质量随视频变化。因此，我们通过引入深度运动先验，识别视频中高质量的关键帧掩码，建立时间一致性。稀疏的关键帧伪标注被用于训练分割模型以实现隐式掩码传播，为此我们提出了基于时序丢弃损失的稀疏到稠密蒸馏方法。在最终模型使用生成的稠密标签集进行训练后，该方法在多个基准上均超越了当前最优性能。
原文摘要 · Abstract (English)
In recent years, the state-of-the-art in unsupervised video instance segmentation has heavily relied on synthetic video data, generated from object-centric image datasets such as ImageNet. However, video synthesis by artificially shifting and scaling image instance masks fails to accurately model realistic motion in videos, such as perspective changes, movement by parts of one or multiple instances, or camera motion. To tackle this issue, we propose an unsupervised video instance segmentation model trained exclusively on real video data. We start from unsupervised instance segmentation masks on individual video frames. However, these single-frame segmentations exhibit temporal noise and their quality varies through the video. Therefore, we establish temporal coherence by identifying high-quality keymasks in the video by leveraging deep motion priors. The sparse keymask pseudo-annotations are then used to train a segmentation model for implicit mask propagation, for which we propose a Sparse-To-Dense Distillation approach aided by a Temporal DropLoss. After training the final model on the resulting dense labelset, our approach outperforms the current state-of-the-art across various benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。