arXiv:2507.09217cs.CV2025-07

提出在线点跟踪新模型Track-On,实现无未来帧下的长期精准追踪。

Online Long-term Point Tracking in the Foundation Model Era

  • 基于变压器架构,逐帧处理视频,以点为查询实现在线追踪。
  • 在7个公开数据集上刷新纪录,验证了无未来信息下的可行性。
  • 适合机器人、AR等需实时决策的场景,尤其看重视角不变性。

点追踪旨在识别视频帧中同一物理点,提供运动的几何感知表示,广泛应用于机器人与增强现实等领域,支持动态环境的精准建模。现有长时追踪方法多为离线设置,可利用未来帧优化预测并恢复遮挡。但在实际场景中,如流媒体和具身智能,需在线因果推理——仅依赖当前及历史帧做即时决策。在此约束下,视角不变性至关重要。视觉基础模型虽缺乏独立的时间推理能力,但其在大规模数据上训练所得的鲁棒几何表征,可作为追踪流水线的空间特征增强。本论文研究在线长时点追踪问题,即在无未来帧或滑动窗口的前提下逐帧处理。实验表明,视觉基础模型可作为有效初始化并融入追踪流程。然而,为实现真正的在线长时追踪,仍需专用设计:在因果条件下维持时间一致性,依赖记忆机制跨帧传递外观与上下文。为此,本文提出Track-On,一种基于变压器的模型,将每个追踪点视为查询,逐帧处理视频。Track-On在七个公开基准上达到新最优,证明了无未来信息下长时追踪的可行性。

原文摘要 · Abstract (English)

Point tracking aims to identify the same physical point across video frames and serves as a geometry-aware representation of motion. This representation supports a wide range of applications, from robotics to augmented reality, by enabling accurate modeling of dynamic environments. Most existing long-term tracking approaches operate in an offline setting, where future frames are available to refine predictions and recover from occlusions. However, real-world scenarios often demand online predictions: the model must operate causally, using only current and past frames. This constraint is critical in streaming video and embodied AI, where decisions must be made immediately based on past observations. Under such constraints, viewpoint invariance becomes essential. Visual foundation models, trained on diverse large-scale datasets, offer the potential for robust geometric representations. While they lack temporal reasoning on their own, they can be integrated into tracking pipelines to enrich spatial features. In this thesis, we address the problem of long-term point tracking in an online setting, where frames are processed sequentially without access to future information or sliding windows. We begin by evaluating the suitability of visual foundation models for this task and find that they can serve as useful initializations and be integrated into tracking pipelines. However, to enable long-term tracking in an online setting, a dedicated design is still required. In particular, maintaining coherence over time in this causal regime requires memory to propagate appearance and context across frames. To address this, we introduce Track-On, a transformer-based model that treats each tracked point as a query and processes video frames one at a time. Track-On sets a new state of the art across seven public benchmarks, demonstrating the feasibility of long-term tracking without future access.

点追踪在线学习视觉基础模型变压器

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。