arXiv:2509.00527cs.CV2025-09ICCV被引 2

用语言模型引导特征解耦,解决增量分割中旧知识丢失问题

Learning Yourself: Class-Incremental Semantic Segmentation with Language-Inspired Bootstrapped Disentanglement

  • 用预训练语言模型的语义先验指导新类特征解耦
  • 在Pascal VOC和ADE20k上达到当前最优,多步增量下优势明显
  • 适合关注持续学习与语义理解融合的研究者

类增量语义分割(CISS)需在不遗忘旧类的前提下持续学习新类别。我们发现主流方法存在根本性挑战:灾难性语义纠缠,包括由语义错位引发的原型-特征纠缠,以及因数据动态演化导致的背景-增量纠缠。现有视觉特征学习方法缺乏有效区分信号,引入大量噪声。为此,我们提出语言启发的自举解耦框架(LBD),利用CLIP等预训练视觉-语言模型的先验语义,通过语言引导原型解耦和流形互惠背景解耦实现特征分离。前者将手工文本特征视为拓扑模板指导新原型解耦,后者采用可学习原型与掩码池化监督实现背景增量类解耦。结合软提示调优与编码器适配修改,进一步缩小了CLIP在密集与稀疏任务间的性能差距,在Pascal VOC和ADE20k上均取得领先结果,尤其在多步增量场景表现优异。

原文摘要 · Abstract (English)

Class-Incremental Semantic Segmentation (CISS) requires continuous learning of newly introduced classes while retaining knowledge of past classes. By abstracting mainstream methods into two stages (visual feature extraction and prototype-feature matching), we identify a more fundamental challenge termed catastrophic semantic entanglement. This phenomenon involves Prototype-Feature Entanglement caused by semantic misalignment during the incremental process, and Background-Increment Entanglement due to dynamic data evolution. Existing techniques, which rely on visual feature learning without sufficient cues to distinguish targets, introduce significant noise and errors. To address these issues, we introduce a Language-inspired Bootstrapped Disentanglement framework (LBD). We leverage the prior class semantics of pre-trained visual-language models (e.g., CLIP) to guide the model in autonomously disentangling features through Language-guided Prototypical Disentanglement and Manifold Mutual Background Disentanglement. The former guides the disentangling of new prototypes by treating hand-crafted text features as topological templates, while the latter employs multiple learnable prototypes and mask-pooling-based supervision for background-incremental class disentanglement. By incorporating soft prompt tuning and encoder adaptation modifications, we further bridge the capability gap of CLIP between dense and sparse tasks, achieving state-of-the-art performance on both Pascal VOC and ADE20k, particularly in multi-step scenarios.

增量学习语义分割视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。