arXiv:2509.22645cs.CVcs.AI2025-09被引 3

用分层语义描述增强CLIP,提升增量学习的识别精度

HERMAN: Hierarchical Representation Matching for CLIP-based Class-Incremental Learning

  • 用大模型生成分层文本描述,显式建模视觉概念层级
  • 在多个基准上实现当前最优性能,有效缓解遗忘问题
  • 适合需要持续学习新类别的视觉系统开发者

类别增量学习(CIL)旨在使模型具备持续适应数据流的能力。近期预训练视觉-语言模型(如CLIP)为此任务提供了强大基础。然而,现有方法通常依赖简单模板(如"一张[类别]的照片"),忽略了视觉概念的层次结构。例如,区分"猫"与"汽车"依赖粗粒度线索,而区分"猫"与"狮子"则需细粒度特征。同时,当前CLIP的特征映射仅使用最后一层表示,忽视了早期层中的层次信息。本文提出面向CLIP的分层表示匹配方法HERMAN:利用大模型递归生成具有区分性的文本描述,扩充语义空间并显式引入层次线索;这些描述按语义层次匹配,并根据任务需求自适应路由,从而实现精确区分并缓解增量学习中的灾难性遗忘。在多个基准上的实验表明,该方法持续达到领先性能。

原文摘要 · Abstract (English)

Class-Incremental Learning (CIL) aims to endow models with the ability to continuously adapt to evolving data streams. Recent advances in pre-trained vision-language models (e.g., CLIP) provide a powerful foundation for this task. However, existing approaches often rely on simplistic templates, such as "a photo of a [CLASS]", which overlook the hierarchical nature of visual concepts. For example, recognizing "cat" versus "car" depends on coarse-grained cues, while distinguishing "cat" from "lion" requires fine-grained details. Similarly, the current feature mapping in CLIP relies solely on the representation from the last layer, neglecting the hierarchical information contained in earlier layers. In this work, we introduce HiErarchical Representation MAtchiNg (HERMAN) for CLIP-based CIL. Our approach leverages LLMs to recursively generate discriminative textual descriptors, thereby augmenting the semantic space with explicit hierarchical cues. These descriptors are matched to different levels of the semantic hierarchy and adaptively routed based on task-specific requirements, enabling precise discrimination while alleviating catastrophic forgetting in incremental tasks. Extensive experiments on multiple benchmarks demonstrate that our method consistently achieves state-of-the-art performance.

增量学习CLIP分层表示视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。