将视频任意点追踪转化为下一个词预测，实现高效在线跟踪。
TAPNext: Tracking Any Point (TAP) as Next Token Prediction
- 把追踪问题转为序列掩码解码，无需复杂先验假设。
- 在多个数据集上达到在线与离线追踪新最优性能。
- 自然涌现出常见追踪启发式，适合追求简洁高效的场景。
视频中任意点的追踪（TAP）是计算机视觉中的挑战性任务，广泛应用于机器人、视频编辑和三维重建。现有方法依赖复杂的追踪专用归纳偏置和启发式规则，限制了通用性和可扩展性。为此，我们提出TAPNext，将TAP建模为序列掩码词元解码问题。该模型为因果结构，支持纯在线追踪，去除了追踪专用的归纳偏置。这使得TAPNext具备极低延迟，并消除了许多先进追踪器所需的时序窗口限制。尽管结构简单，TAPNext在在线与离线追踪中均达到新最优性能。此外，我们发现许多广泛使用的追踪启发式可通过端到端训练自然涌现。TAPNext模型与代码见https://tap-next.github.io/。
原文摘要 · Abstract (English)
Tracking Any Point (TAP) in a video is a challenging computer vision problem with many demonstrated applications in robotics, video editing, and 3D reconstruction. Existing methods for TAP rely heavily on complex tracking-specific inductive biases and heuristics, limiting their generality and potential for scaling. To address these challenges, we present TAPNext, a new approach that casts TAP as sequential masked token decoding. Our model is causal, tracks in a purely online fashion, and removes tracking-specific inductive biases. This enables TAPNext to run with minimal latency, and removes the temporal windowing required by many existing state of art trackers. Despite its simplicity, TAPNext achieves a new state-of-the-art tracking performance among both online and offline trackers. Finally, we present evidence that many widely used tracking heuristics emerge naturally in TAPNext through end-to-end training. The TAPNext model and code can be found at https://tap-next.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。