用图像扩散模型实现视频零样本目标跟踪，无需训练即可精准定位物体。
Image Diffusion Models Exhibit Emergent Temporal Propagation in Videos
- 将扩散模型的注意力图视为语义传播核，跨帧扩展出时间传播机制。
- 在多个标准数据集上达到当前最优的零样本视频目标分割性能。
- 适合需要快速部署、无标注数据场景下的视频理解任务。
图像扩散模型虽为图像生成设计，却隐含丰富的语义结构，可支持识别与定位等任务。本文发现其自注意力图可重新解释为语义标签传播核，实现图像区域内像素级对应。将该机制拓展至视频帧间，形成时间传播核，从而通过分割实现零样本目标跟踪。进一步验证了测试时优化策略（如DDIM反演、文本反演、自适应头加权）对特征适配与传播鲁棒性的提升效果。基于此，提出DRIFT框架，利用预训练图像扩散模型结合SAM引导的掩码精修，在标准视频对象分割基准上实现零样本领先性能。
原文摘要 · Abstract (English)
Image diffusion models, though originally developed for image generation, implicitly capture rich semantic structures that enable various recognition and localization tasks beyond synthesis. In this work, we investigate their self-attention maps can be reinterpreted as semantic label propagation kernels, providing robust pixel-level correspondences between relevant image regions. Extending this mechanism across frames yields a temporal propagation kernel that enables zero-shot object tracking via segmentation in videos. We further demonstrate the effectiveness of test-time optimization strategies-DDIM inversion, textual inversion, and adaptive head weighting-in adapting diffusion features for robust and consistent label propagation. Building on these findings, we introduce DRIFT, a framework for object tracking in videos leveraging a pretrained image diffusion model with SAM-guided mask refinement, achieving state-of-the-art zero-shot performance on standard video object segmentation benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。