高效高分辨率点追踪，一次算出一帧对数百帧的密集运动场。
AllTracker: Efficient Dense Point Tracking at High Resolution
- 基于低分辨率网格迭代推演，结合2D卷积与像素对齐注意力跨空间时空传播信息。
- 在768x1024分辨率下实现顶尖追踪精度，仅需1600万参数，40G显存运行。
- 可联合训练光流与点追踪数据集，适合需要长时序高精度运动建模的场景。
我们提出AllTracker：一种通过估算查询帧与视频中其他所有帧之间的运动场来实现长程点追踪的模型。与现有方法不同，该方法可生成高分辨率、稠密（全像素）的对应关系场，可视化为运动图。不同于传统光流方法仅预测相邻帧，本方法可将一帧与后续数百帧建立对应。我们设计新架构，融合光流与点追踪技术：在低分辨率网格上进行迭代推理，通过二维卷积实现空间信息传播，利用像素对齐注意力实现时间信息传播。模型快速且参数高效（1600万参数），在768x1024高分辨率下达到当前最佳追踪精度（使用40G GPU）。设计优势在于可联合训练光流与点追踪数据集，实验表明这对性能至关重要。我们提供了详尽的消融实验，明确关键架构与训练细节。代码与模型权重见https://alltracker.github.io。
原文摘要 · Abstract (English)
We introduce AllTracker: a model that estimates long-range point tracks by way of estimating the flow field between a query frame and every other frame of a video. Unlike existing point tracking methods, our approach delivers high-resolution and dense (all-pixel) correspondence fields, which can be visualized as flow maps. Unlike existing optical flow methods, our approach corresponds one frame to hundreds of subsequent frames, rather than just the next frame. We develop a new architecture for this task, blending techniques from existing work in optical flow and point tracking: the model performs iterative inference on low-resolution grids of correspondence estimates, propagating information spatially via 2D convolution layers, and propagating information temporally via pixel-aligned attention layers. The model is fast and parameter-efficient (16 million parameters), and delivers state-of-the-art point tracking accuracy at high resolution (i.e., tracking 768x1024 pixels, on a 40G GPU). A benefit of our design is that we can train jointly on optical flow datasets and point tracking datasets, and we find that doing so is crucial for top performance. We provide an extensive ablation study on our architecture details and training recipe, making it clear which details matter most. Our code and model weights are available at https://alltracker.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。