用记忆模块实现高效在线点追踪，实时稳定追踪长视频中的关键点。
Track-On: Transformer-based Online Point Tracking with Memory
- 基于因果Transformer结构，利用空间与上下文记忆模块捕捉时序信息。
- 在7个数据集上达到新SOTA，TAP-Vid上性能超越现有离线方法。
- 适合实时视频流场景，适用于自动驾驶、机器人视觉等应用。
本文研究长期点追踪问题，要求在视频多帧间保持点的连续识别，即使面对外观、光照、视角变化及遮挡。针对在线逐帧处理需求，提出Track-On模型，一种基于Transformer的轻量级在线长时点追踪方法。不同于依赖完整时序建模的先前方法，该模型采用因果处理机制，不使用未来帧信息，并通过空间记忆和上下文记忆模块捕捉长期依赖，维持追踪稳定性。推理阶段结合补丁分类与精修策略，实现高精度对应匹配。在七个数据集上的实验表明，Track-On在在线模型中达到新最佳性能，在TAP-Vid等基准上表现优于或媲美离线方法。本方法为真实场景下的实时追踪提供了鲁棒且可扩展的解决方案。
原文摘要 · Abstract (English)
In this paper, we consider the problem of long-term point tracking, which requires consistent identification of points across multiple frames in a video, despite changes in appearance, lighting, perspective, and occlusions. We target online tracking on a frame-by-frame basis, making it suitable for real-world, streaming scenarios. Specifically, we introduce Track-On, a simple transformer-based model designed for online long-term point tracking. Unlike prior methods that depend on full temporal modeling, our model processes video frames causally without access to future frames, leveraging two memory modules -- spatial memory and context memory -- to capture temporal information and maintain reliable point tracking over long time horizons. At inference time, it employs patch classification and refinement to identify correspondences and track points with high accuracy. Through extensive experiments, we demonstrate that Track-On sets a new state-of-the-art for online models and delivers superior or competitive results compared to offline approaches on seven datasets, including the TAP-Vid benchmark. Our method offers a robust and scalable solution for real-time tracking in diverse applications. Project page: https://kuis-ai.github.io/track_on
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。