无需训练，用区域表示实现长视频视觉查询定位
RELOCATE: A Simple Training-Free Baseline for Visual Query Localization Using Region-Based Representations
- 基于预训练模型的区域表示，不依赖任务特定训练
- 在Ego4D数据集上相对提升49%的时空平均精度
- 适合快速部署于长视频定位任务，尤其小目标和遮挡场景
我们提出RELOCATE，一种无需训练的基准方法，用于解决长视频中视觉查询定位这一挑战性任务。为避免任务特定训练并高效处理长视频，RELOCATE利用预训练视觉模型生成的区域表示。整体流程遵循经典目标定位思路：(1) 检测每帧中的所有物体；(2) 将物体与给定查询比较，选取最相似者；(3) 通过双向跟踪获得时空响应。针对小物体、杂乱场景、部分可见及外观变化等问题，我们提出了关键改进：精修选中物体以提升定位精度，并生成额外视觉查询以捕捉视觉变体。在具有挑战性的Ego4D Visual Query 2D Localization数据集上评估，该方法建立了一个新基准，相比先前任务特定方法在时空平均精度上提升了49%（相对提升）。
原文摘要 · Abstract (English)
We present RELOCATE, a simple training-free baseline designed to perform the challenging task of visual query localization in long videos. To eliminate the need for task-specific training and efficiently handle long videos, RELOCATE leverages a region-based representation derived from pretrained vision models. At a high level, it follows the classic object localization approach: (1) identify all objects in each video frame, (2) compare the objects with the given query and select the most similar ones, and (3) perform bidirectional tracking to get a spatio-temporal response. However, we propose some key enhancements to handle small objects, cluttered scenes, partial visibility, and varying appearances. Notably, we refine the selected objects for accurate localization and generate additional visual queries to capture visual variations. We evaluate RELOCATE on the challenging Ego4D Visual Query 2D Localization dataset, establishing a new baseline that outperforms prior task-specific methods by 49% (relative improvement) in spatio-temporal average precision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。