arXiv:2501.12218cs.CV2025-01CVPR被引 15

提出可直接追踪点的时序感知特征模型,无需后期修正

Exploring Temporally-Aware Features for Point Tracking

  • 用预训练DINOv2+时序适配器构建带时间感知的特征主干
  • 在无修正阶段下达到顶尖性能,速度远超现有方法
  • 适合需要高效精准点追踪的应用场景

视频中的点追踪是机器人、视频编辑等任务的基础。尽管许多视觉任务受益于预训练特征主干以提升泛化能力,点追踪仍主要依赖从头训练的简单主干,且多基于合成数据,可能影响真实场景下的鲁棒性。此外,点追踪需具备时序感知以保证帧间一致性,但当前多数方法采用两阶段流程:先粗略预测,再通过修正阶段注入时序信息并纠正错误,该方式计算开销大且冗余。本文提出Chrono,一种专为点追踪设计、内置时序感知能力的特征主干。它利用自监督学习模型DINOv2的预训练表示,并通过时序适配器增强,有效捕捉长期时序上下文,实现无需修正阶段的精确预测。实验表明,在TAP-Vid-DAVIS和TAP-Vid-Kinetics数据集上,Chrono在不使用修正器的情况下,优于常见特征主干及DINOv2,达到当前最优性能,且效率显著。

原文摘要 · Abstract (English)

Point tracking in videos is a fundamental task with applications in robotics, video editing, and more. While many vision tasks benefit from pre-trained feature backbones to improve generalizability, point tracking has primarily relied on simpler backbones trained from scratch on synthetic data, which may limit robustness in real-world scenarios. Additionally, point tracking requires temporal awareness to ensure coherence across frames, but using temporally-aware features is still underexplored. Most current methods often employ a two-stage process: an initial coarse prediction followed by a refinement stage to inject temporal information and correct errors from the coarse stage. These approach, however, is computationally expensive and potentially redundant if the feature backbone itself captures sufficient temporal information. In this work, we introduce Chrono, a feature backbone specifically designed for point tracking with built-in temporal awareness. Leveraging pre-trained representations from self-supervised learner DINOv2 and enhanced with a temporal adapter, Chrono effectively captures long-term temporal context, enabling precise prediction even without the refinement stage. Experimental results demonstrate that Chrono achieves state-of-the-art performance in a refiner-free setting on the TAP-Vid-DAVIS and TAP-Vid-Kinetics datasets, among common feature backbones used in point tracking as well as DINOv2, with exceptional efficiency. Project page: https://cvlab-kaist.github.io/Chrono/

点追踪时序感知特征主干DINOv2

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。