CRISP通过对比残差注入与语义提示,解决视频实例分割的持续学习难题。
CRISP: Contrastive Residual Injection and Semantic Prompting for Continual Video Instance Segmentation
- 引入对比残差注入,增强实例跟踪与查询特异性。
- 在YouTube-VIS-2019/2021上实现更优分割与分类性能,避免灾难性遗忘。
- 适合需要长期视频分割且关注类别与任务连续性的研究者。
持续视频实例分割需兼具吸收新类别与保留旧知识的能力,同时维持帧间时序一致性。本文提出对比残差注入与语义提示(CRISP),首次针对实例、类别和任务层面的混淆问题进行系统建模。针对实例级学习,构建实例相关性损失,强化当前任务查询与先前查询空间的关联性并提升其特异性。针对类别级学习,设计自适应残差语义提示(ARSP)框架,利用类别文本生成可学习的语义残差提示池,并通过动态查询-提示匹配机制建立映射关系;同时引入基于对比学习的语义一致性损失,保持增量训练中对象查询与残差提示间的语义连贯性。针对任务级学习,提出简洁高效的增量提示初始化策略,确保查询空间跨任务的关联性。在YouTube-VIS-2019与YouTube-VIS-2021数据集上的大量实验表明,CRISP显著优于现有方法,在长期持续视频实例分割任务中有效缓解灾难性遗忘,显著提升分割与分类性能。代码已开源:https://github.com/01upup10/CRISP。
原文摘要 · Abstract (English)
Continual video instance segmentation demands both the plasticity to absorb new object categories and the stability to retain previously learned ones, all while preserving temporal consistency across frames. In this work, we introduce Contrastive Residual Injection and Semantic Prompting (CRISP), an earlier attempt tailored to address the instance-wise, category-wise, and task-wise confusion in continual video instance segmentation. For instance-wise learning, we model instance tracking and construct instance correlation loss, which emphasizes the correlation with the prior query space while strengthening the specificity of the current task query. For category-wise learning, we build an adaptive residual semantic prompt (ARSP) learning framework, which constructs a learnable semantic residual prompt pool generated by category text and uses an adjustive query-prompt matching mechanism to build a mapping relationship between the query of the current task and the semantic residual prompt. Meanwhile, a semantic consistency loss based on the contrastive learning is introduced to maintain semantic coherence between object queries and residual prompts during incremental training. For task-wise learning, to ensure the correlation at the inter-task level within the query space, we introduce a concise yet powerful initialization strategy for incremental prompts. Extensive experiments on YouTube-VIS-2019 and YouTube-VIS-2021 datasets demonstrate that CRISP significantly outperforms existing continual segmentation methods in the long-term continual video instance segmentation task, avoiding catastrophic forgetting and effectively improving segmentation and classification performance. The code is available at https://github.com/01upup10/CRISP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。