让视觉语言模型在持续学习中自动增强零样本分类能力。
Modular Embedding Recomposition for Incremental Learning
- 用多个专精于单类的文本专家构建基础库，动态组合生成新类原型。
- 在14个数据集上验证，显著提升未见类的零样本分类准确率。
- 适合需要持续学习新类且不重新训练的场景，如智能监控、医疗图像分析。
预训练视觉语言模型（VLM）的出现极大推动了持续学习（CL）的发展，主要得益于其零样本分类能力。这一特性使VLM在真实场景中表现优异，无需适应即可对新类别进行有效识别。然而，当下游任务与预训练领域差异较大时，微调仍必不可少。现有方法多聚焦于在增量微调过程中保留VLM的零样本能力。本文提出更进一步的思路：将能力保留转化为能力增强。所提方法名为模块化嵌入重构（MoDER），构建一个模块化框架，训练多个专精于单一已见类的文本专家，并将其存储于基础知识库中。推理时，针对每个未见类，从知识库中检索相关专家并组合生成优化后的类别原型，从而提升分类性能。在两类主流零样本持续学习协议（Class-IL 和 MTIL）共14个数据集上验证了该方法的有效性。代码已公开于 https://github.com/aimagelab/mammoth。
原文摘要 · Abstract (English)
The advent of pre-trained Vision-Language Models (VLMs) has significantly transformed Continual Learning (CL), mainly due to their zero-shot classification abilities. Such proficiency makes VLMs well-suited for real-world applications, enabling robust performance on novel unseen classes without requiring adaptation. However, fine-tuning remains essential when downstream tasks deviate significantly from the pre-training domain. Prior CL approaches primarily focus on preserving the zero-shot capabilities of VLMs during incremental fine-tuning on a downstream task. We take a step further by devising an approach that transforms preservation into enhancement of the zero-shot capabilities of VLMs. Our approach, named MoDular Embedding Recomposition (MoDER), introduces a modular framework that trains multiple textual experts, each specialized in a single seen class, and stores them in a foundational hub. At inference time, for each unseen class, we query the hub and compose the retrieved experts to synthesize a refined prototype that improves classification. We show the effectiveness of our method across two popular zero-shot incremental protocols, Class-IL and MTIL, comprising a total of 14 datasets. The codebase is available at https://github.com/aimagelab/mammoth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。