arXiv:2512.04619cs.CV2025-12被引 3

利用视频扩散模型的先验知识,实现无需标注数据的零样本点跟踪。

Denoise to Track: Harnessing Video Diffusion Priors for Robust Correspondence

  • 通过分析注意力头功能,选择关键低频特征提升匹配精度。
  • 在TAP-Vid上达到当前最佳零样本跟踪效果,接近有监督方法性能。
  • 适合需要少标注或无标注的视觉跟踪任务研究者使用。

本文提出HeFT(Head-Frequency Tracker),一种基于预训练视频扩散模型视觉先验的零样本点跟踪框架。为理解其时空信息编码机制,我们分析了视频扩散变换器(VDiT)的内部表征,发现注意力头作为最小功能单元,分别专用于匹配、语义理解与位置编码;同时发现低频成分对建立对应关系至关重要,而高频成分易引入噪声。基于此,我们设计了头与频率感知的特征选择策略,联合选取最有效注意力头与低频成分以增强跟踪性能。具体方法包括单步去噪提取判别性特征、特征选择,以及结合前向-后向一致性检查的软argmax定位。在TAP-Vid基准上的大量实验表明,HeFT实现了当前最优的零样本跟踪性能,接近有监督方法水平,且无需标注训练数据。本工作进一步验证了视频扩散模型作为通用视觉基础模型的巨大潜力,推动统一视觉基础模型的发展。

原文摘要 · Abstract (English)

In this work, we introduce HeFT (Head-Frequency Tracker), a zero-shot point tracking framework that leverages the visual priors of pretrained video diffusion models. To better understand how they encode spatiotemporal information, we analyze the internal representations of Video Diffusion Transformer (VDiT). Our analysis reveals that attention heads act as minimal functional units with distinct specializations for matching, semantic understanding, and positional encoding. Additionally, we find that the low-frequency components in VDiT features are crucial for establishing correspondences, whereas the high-frequency components tend to introduce noise. Building on these insights, we propose a head- and frequency-aware feature selection strategy that jointly selects the most informative attention head and low-frequency components to enhance tracking performance. Specifically, our method extracts discriminative features through single-step denoising, applies feature selection, and employs soft-argmax localization with forward-backward consistency checks for correspondence estimation. Extensive experiments on TAP-Vid benchmarks demonstrate that HeFT achieves state-of-the-art zero-shot tracking performance, approaching the accuracy of supervised methods while eliminating the need for annotated training data. Our work further underscores the promise of video diffusion models as powerful foundation models for a wide range of downstream tasks, paving the way toward unified visual foundation models.

视频跟踪扩散模型零样本特征选择

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。