为视频实例分割设计新预训练框架，提升复杂视频中物体关系建模能力。
A Temporal Modeling Framework for Video Pre-Training on Video Instance Segmentation
- 用一致伪视频增强生成多样化训练样本，保持帧间实例一致性。
- 引入多尺度时序模块，通过自/交叉注意力捕捉长短时序关系，性能提升4.0%。
- 无需修改模型结构，可适配多种分割方法，适合视频理解研究者。
当前视频实例分割（VIS）方法普遍采用图像预训练后微调至视频的范式，但预训练模型缺乏时序知识，导致预训练与微调阶段存在领域差距，影响性能。为此，本文提出一种新型视频预训练方法，以增强复杂视频中实例关系建模能力。核心创新在于缩小预训练与微调阶段的差异：首先引入一致伪视频增强，在生成多样伪视频样本的同时保持帧间实例一致性；其次设计多尺度时序模块，通过短程与长程自注意力和交叉注意力提升模型对时序关系的建模能力。该方法不依赖特定模型架构，可无缝集成于多种VIS方法。在主流VIS基准测试中，本方法持续优于现有最佳方法，在挑战性数据集OVIS上实现平均精度提升4.0%。
原文摘要 · Abstract (English)
Contemporary Video Instance Segmentation (VIS) methods typically adhere to a pre-train then fine-tune regime, where a segmentation model trained on images is fine-tuned on videos. However, the lack of temporal knowledge in the pre-trained model introduces a domain gap which may adversely affect the VIS performance. To effectively bridge this gap, we present a novel video pre-training approach to enhance VIS models, especially for videos with intricate instance relationships. Our crucial innovation focuses on reducing disparities between the pre-training and fine-tuning stages. Specifically, we first introduce consistent pseudo-video augmentations to create diverse pseudo-video samples for pre-training while maintaining the instance consistency across frames. Then, we incorporate a multi-scale temporal module to enhance the model's ability to model temporal relations through self- and cross-attention at short- and long-term temporal spans. Our approach does not set constraints on model architecture and can integrate seamlessly with various VIS methods. Experiment results on commonly adopted VIS benchmarks show that our method consistently outperforms state-of-the-art methods. Our approach achieves a notable 4.0% increase in average precision on the challenging OVIS dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。