通过查询匹配与特征移位，提升视频语义分割的时序一致性。
Shift and matching queries for video semantic segmentation
- 先匹配查询再做特征移位，确保跨帧查询对应同一对象。
- 在CityScapes-VPS和VSPW上显著优于基线模型。
- 可复用预训练图像分割权重，高效且效果好。
视频分割是热门任务，但将图像分割模型逐帧应用到视频中无法保证时序一致性。本文提出一种方法，通过特征移位与查询匹配，将基于查询的图像分割模型扩展至视频领域。该方法采用查询架构，解码后的查询代表分割掩码。为确保移位后查询仍对应同一掩码，需先进行查询匹配。在CityScapes-VPS和VSPW数据集上的实验表明，该方法显著优于基线，有效提升了分割质量，同时高效复用预训练权重。
原文摘要 · Abstract (English)
Video segmentation is a popular task, but applying image segmentation models frame-by-frame to videos does not preserve temporal consistency. In this paper, we propose a method to extend a query-based image segmentation model to video using feature shift and query matching. The method uses a query-based architecture, where decoded queries represent segmentation masks. These queries should be matched before performing the feature shift to ensure that the shifted queries represent the same mask across different frames. Experimental results on CityScapes-VPS and VSPW show significant improvements from the baselines, highlighting the method's effectiveness in enhancing segmentation quality while efficiently reusing pre-trained weights.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。