同时利用外观与运动线索,提升视频全景跟踪精度。
Learning Appearance and Motion Cues for Panoptic Tracking
- 设计双分支网络,分别提取外观和运动特征。
- 在两个基准数据集上达到当前最优跟踪准确率。
- 适合需要长期目标识别的机器人场景研究。
全景跟踪通过整合全景分割中的实例跟踪,实现视频的像素级场景理解,为机器人在动态环境中提供时空认知能力。本文提出一种新方法,同步捕捉通用语义信息及实例级外观与运动特征。不同于忽略动态特性的现有方法,本方法通过专用网络头同时利用外观与运动线索。这些相互关联的头采用多尺度可变形卷积,结合语义上下文与运动增强的外观特征来推理场景运动偏移,学习追踪嵌入。此外,引入新颖的两步融合模块:先匹配当前帧与前一帧的实例,再用运动增强的外观嵌入优化关联,提升复杂场景下的鲁棒性。在两个基准数据集上的大量实验表明,所提 etname 模型在全景跟踪精度上达到当前最优,显著优于先前方法在长时间保持目标身份的能力。代码已公开于 http://panoptictracking.cs.uni-freiburg.de。
原文摘要 · Abstract (English)
Panoptic tracking enables pixel-level scene interpretation of videos by integrating instance tracking in panoptic segmentation. This provides robots with a spatio-temporal understanding of the environment, an essential attribute for their operation in dynamic environments. In this paper, we propose a novel approach for panoptic tracking that simultaneously captures general semantic information and instance-specific appearance and motion features. Unlike existing methods that overlook dynamic scene attributes, our approach leverages both appearance and motion cues through dedicated network heads. These interconnected heads employ multi-scale deformable convolutions that reason about scene motion offsets with semantic context and motion-enhanced appearance features to learn tracking embeddings. Furthermore, we introduce a novel two-step fusion module that integrates the outputs from both heads by first matching instances from the current time step with propagated instances from previous time steps and subsequently refines associations using motion-enhanced appearance embeddings, improving robustness in challenging scenarios. Extensive evaluations of our proposed \netname model on two benchmark datasets demonstrate that it achieves state-of-the-art performance in panoptic tracking accuracy, surpassing prior methods in maintaining object identities over time. To facilitate future research, we make the code available at http://panoptictracking.cs.uni-freiburg.de
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。