提出局部全局查询对齐机制,提升视频实例分割的时序一致性。
Local2Global query Alignment for Video Instance Segmentation

- 引入局部与全局查询,分别捕捉帧内特征和历史时空信息。
- 在Youtube-VIS-19/21上达54.3/49.4 AP,OVIS上达37.0 AP。
- 纯在线训练,无需复杂记忆或启发式设计,适合实时应用。
在线视频分割方法在处理长序列和渐变变化方面表现优异,适用于真实场景。但时序一致性仍面临挑战,尤其在在线传播中噪声累积、遮挡突变和场景切换时。本文提出Local2Global框架,基于DETR的查询传播结构,引入两类新查询:(1) 局部查询,从每帧提取初始对象特定空间特征;(2) 全局查询,包含历史时空表征。提出L2G-aligner——一种轻量级Transformer解码器,实现局部与全局查询的早期对齐。该机制使模型能有效利用当前帧信息并保持时序一致,实现帧间平滑过渡。且L2G-aligner直接嵌入分割模型,无需额外复杂启发式或记忆模块。在多个挑战性VIS和VPS数据集上实验表明,本方法以简单在线训练超越现有基准,无需复杂组件。例如,在Youtube-VIS-19/-21上分别取得54.3和49.4 AP,OVIS上达37.0 AP(ResNet-50骨干)。
原文摘要 · Abstract (English)
Online video segmentation methods excel at handling long sequences and capturing gradual changes, making them ideal for real-world applications. However, achieving temporally consistent predictions remains a challenge, especially with gradual accumulation of noise or drift in on-line propagation, abrupt occlusions and scene transitions. This paper introduces Local2Global, an online framework, for video instance segmentation, exhibiting state-of-the-art performance with simple baseline and training purely in online fashion. Leveraging the DETR-based query propagation framework, we introduce two novel sets of queries:(1) local queries that capture initial object-specific spatial features from each frame and (2) global queries containing past spatio-temporal representations. We propose the L2G-aligner, a novel lightweight transformer decoder, to facilitate an early alignment between local and global queries. This alignment allows our model to effectively utilize current frame information while maintaining temporal consistency, producing a smooth transition between frames. Furthermore, L2G-aligner is integrated within the segmentation model, without relying on additional complex heuristics, or memory mechanisms. Extensive experiments across various challenging VIS and VPS datasets showcase the superiority of our method with simple online training, surpassing current benchmarks without bells and rings. For instance, we achieve 54.3 and 49.4 AP on Youtube-VIS-19/-21 datasets and 37.0 AP on OVIS dataset respectively withthe ResNet-50 backbone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。