提出分层视觉提示学习,解决视频实例分割持续学习中的遗忘问题。
Hierarchical Visual Prompt Learning for Continual Video Instance Segmentation
- 设计帧级与视频级任务特定提示,分层缓解遗忘
- 通过正交梯度投影保留旧类别特征,避免灾难性遗忘
- 适合需要持续学习新类别的视频分割场景
视频实例分割(VIS)在跨帧跟踪和分割物体实例方面受到广泛关注。然而,现有方法通常假设物体类别在时间上保持不变,且在持续学习新类别时会严重遗忘旧类别。为此,本文提出一种新型分层视觉提示学习(HVPL)模型,从帧级和视频级两个层面缓解灾难性遗忘。为减轻帧级遗忘,设计任务特定的帧提示与正交梯度修正(OGC)模块,通过将梯度投影至旧类别的正交特征空间,使帧提示在单帧中编码新类别的全局实例信息。为应对视频级遗忘,设计任务特定的视频提示与视频上下文解码器,该解码器先将跨帧的类间结构关系嵌入帧提示特征,再将任务相关的全局视频上下文从帧提示传播至视频提示。大量对比实验表明,所提方法显著优于基线模型。代码已开源。
原文摘要 · Abstract (English)
Video instance segmentation (VIS) has gained significant attention for its capability in tracking and segmenting object instances across video frames. However, most of the existing VIS approaches unrealistically assume that the categories of object instances remain fixed over time. Moreover, they experience catastrophic forgetting of old classes when required to continuously learn object instances belonging to new categories. To resolve these challenges, we develop a novel Hierarchical Visual Prompt Learning (HVPL) model that overcomes catastrophic forgetting of previous categories from both frame-level and video-level perspectives. Specifically, to mitigate forgetting at the frame level, we devise a task-specific frame prompt and an orthogonal gradient correction (OGC) module. The OGC module helps the frame prompt encode task-specific global instance information for new classes in each individual frame by projecting its gradients onto the orthogonal feature space of old classes. Furthermore, to address forgetting at the video level, we design a task-specific video prompt and a video context decoder. This decoder first embeds structural inter-class relationships across frames into the frame prompt features, and then propagates task-specific global video contexts from the frame prompt features to the video prompt. Through rigorous comparisons, our HVPL model proves to be more effective than baseline approaches. The code is available at https://github.com/JiahuaDong/HVPL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。