首个基于事件相机的任意点追踪方法，实现高速高动态场景下的精准跟踪。
ETAP: Event-based Tracking of Any Point
- 融合事件相机高帧率与全局上下文，实现异步稀疏事件数据的鲁棒追踪。
- 在跨数据集测试中平均杰卡德指数领先基线136%，特征追踪任务提升20%。
- 提出新型特征对齐损失，有效应对运动引起的特征变化，适合高速动态场景研究者。
追踪任意点（TAP）将运动估计范式从依赖局部模板的显著点转向利用全局图像上下文追踪任意点。然而，现有研究主要关注正常条件下的精度提升，难以应对复杂光照和高速运动，这受限于传感器性能。本文首次提出基于事件相机的TAP方法，利用事件相机的高时间分辨率和高动态范围，实现鲁棒的高速追踪；同时借助TAP框架中的全局上下文，处理事件数据的异步与稀疏特性。进一步针对运动导致的事件特征变化这一开放挑战，设计了新颖的特征对齐损失，以学习运动鲁棒特征。方法通过新构建的数据生成管道训练，并系统性地评估各项设计选择。实验表明，该方法具备强跨数据集泛化能力，在平均杰卡德指标上比基线提升136%；在标准特征追踪基准上，较此前最佳纯事件方法提升20%，甚至超过此前最佳事件-帧联合方法4.1%。代码已开源。
原文摘要 · Abstract (English)
Tracking any point (TAP) recently shifted the motion estimation paradigm from focusing on individual salient points with local templates to tracking arbitrary points with global image contexts. However, while research has mostly focused on driving the accuracy of models in nominal settings, addressing scenarios with difficult lighting conditions and high-speed motions remains out of reach due to the limitations of the sensor. This work addresses this challenge with the first event camera-based TAP method. It leverages the high temporal resolution and high dynamic range of event cameras for robust high-speed tracking, and the global contexts in TAP methods to handle asynchronous and sparse event measurements. We further extend the TAP framework to handle event feature variations induced by motion -- thereby addressing an open challenge in purely event-based tracking -- with a novel feature-alignment loss which ensures the learning of motion-robust features. Our method is trained with data from a new data generation pipeline and systematically ablated across all design decisions. Our method shows strong cross-dataset generalization and performs 136% better on the average Jaccard metric than the baselines. Moreover, on an established feature tracking benchmark, it achieves a 20% improvement over the previous best event-only method and even surpasses the previous best events-and-frames method by 4.1%. Our code is available at https://github.com/tub-rip/ETAP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。