用扩散模型加追踪机制，实现任意粒度视频目标分割。
SMITE: Segment Me In TimE
- 基于预训练图文扩散模型，结合追踪机制进行视频分割。
- 可在任意粒度下保持标签一致性，优于现有方法。
- 适合需要灵活分割的视频理解任务，如医学或动画分析。
视频中对物体进行分割面临巨大挑战：每个像素需精准标注，且标签在帧间必须保持一致。当分割粒度任意时——即分割数量可自由变化，且仅依赖一两张样本图像定义掩码——问题更加复杂。本文提出一种新方法，利用预训练的文本到图像扩散模型，并引入额外追踪机制，有效应对多种分割场景，实验表明其性能超越当前最优方法。
原文摘要 · Abstract (English)
Segmenting an object in a video presents significant challenges. Each pixel must be accurately labelled, and these labels must remain consistent across frames. The difficulty increases when the segmentation is with arbitrary granularity, meaning the number of segments can vary arbitrarily, and masks are defined based on only one or a few sample images. In this paper, we address this issue by employing a pre-trained text to image diffusion model supplemented with an additional tracking mechanism. We demonstrate that our approach can effectively manage various segmentation scenarios and outperforms state-of-the-art alternatives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。