让追踪模型学会看3D空间关系,提升遮挡和杂乱环境下的追踪精度。
GOT-Edit: Geometry-Aware Generic Object Tracking via Online Model Editing
- 用2D图像推断3D几何信息,实时更新追踪模型。
- 在遮挡和干扰下追踪准确率显著提升,跨场景表现更稳定。
- 适合需要高鲁棒性的视觉追踪应用,如自动驾驶、机器人导航。
人类在2D视频中有效追踪物体依赖于隐含的3D知识和语义推理。相比之下,多数通用物体追踪(GOT)方法仅依赖目标及其周围区域的2D特征,忽略3D几何线索,导致对部分遮挡、干扰及形貌变化敏感。为此,我们提出GOT-Edit,一种基于2D视频流的在线跨模态模型编辑方法,将几何感知线索融入通用追踪器。该方法利用预训练的视觉几何接地变压器(Visual Geometry Grounded Transformer)从少量2D图像中推断几何线索。为实现几何与语义的无缝融合,GOT-Edit执行在线模型编辑,通过零空间约束在更新过程中保留语义区分能力的同时引入几何信息,从而在多种场景下持续提升性能。在多个GOT基准上的实验表明,GOT-Edit在遮挡和杂乱环境下表现出更强的鲁棒性与准确性,树立了结合2D语义与3D几何推理的新范式。
原文摘要 · Abstract (English)
Human perception for effective object tracking in 2D video streams arises from the implicit use of prior 3D knowledge and semantic reasoning. In contrast, most generic object tracking (GOT) methods primarily rely on 2D features of the target and its surroundings, while neglecting 3D geometric cues, making them susceptible to partial occlusion, distractors, and variations in geometry and appearance. To address this limitation, we introduce GOT-Edit, an online cross-modality model editing approach that integrates geometry-aware cues into a generic object tracker from a 2D video stream. Our approach leverages features from a pre-trained Visual Geometry Grounded Transformer to infer geometric cues from only a few 2D images. To address the challenge of seamlessly combining geometry and semantics, GOT-Edit performs online model editing. By leveraging null-space constraints during model updates, it incorporates geometric information while preserving semantic discrimination, yielding consistently better performance across diverse scenarios. Extensive experiments on multiple GOT benchmarks demonstrate that GOT-Edit achieves superior robustness and accuracy, particularly under occlusion and clutter, establishing a new paradigm for combining 2D semantics with 3D geometric reasoning for generic object tracking. The project page is available at https://chenshihfang.github.io/GOT-EDIT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。