LADA通过标签专属记忆单元实现高效持续学习
LADA: Scalable Label-Specific CLIP Adapter for Continual Learning
- 为CLIP图像编码器添加轻量级标签专用记忆单元
- 在多个任务上达到当前最优持续学习性能
- 适合需要稳定增量学习的视觉语言系统
基于CLIP等视觉-语言模型的持续学习为可扩展机器学习系统提供了路径，利用其可迁移表征。现有方法通过在预训练图像编码器中添加多组可学习参数来适应不同任务，每个任务使用部分参数。但推理时需选择对应参数，易出错导致性能下降。为此，本文提出LADA（标签特定适配器）。LADA不将参数按任务划分，而是向冻结的CLIP图像编码器附加轻量级、标签专属的记忆单元，通过聚合与任务无关的知识生成判别性特征。为防止灾难性遗忘，LADA采用特征蒸馏保护已见类别的特征，避免被新类别干扰。由于位置位于图像编码器之后，LADA阻止梯度反传至冻结的CLIP参数，确保高效训练。大量实验表明，LADA在持续学习设置下达到当前最优性能。代码已开源：https://github.com/MaolinLuo/LADA。
原文摘要 · Abstract (English)
Continual learning with vision-language models like CLIP offers a pathway toward scalable machine learning systems by leveraging its transferable representations. Existing CLIP-based methods adapt the pre-trained image encoder by adding multiple sets of learnable parameters, with each task using a partial set of parameters. This requires selecting the expected parameters for input images during inference, which is prone to error that degrades performance. To address this problem, we introduce LADA (Label-specific ADApter). Instead of partitioning parameters across tasks, LADA appends lightweight, label-specific memory units to the frozen CLIP image encoder, enabling discriminative feature generation by aggregating task-agnostic knowledge. To prevent catastrophic forgetting, LADA employs feature distillation for seen classes, preventing their features from being interfered with by new classes. Positioned after the image encoder, LADA prevents gradient flow to the frozen CLIP parameters, ensuring efficient training. Extensive results show that LADA achieves state-of-the-art performance in continual learning settings. The implementation code is available at https://github.com/MaolinLuo/LADA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。