用动态加权损失让CLIP模型小内存下持续学习不遗忘
Memory-Efficient Continual Learning with CLIP Models
- 按类别动态重加权损失，提升小内存下的训练稳定性
- 在CIFAR-100、ImageNet1K和DomainNet上实现低遗忘快速适应
- 适合资源受限场景下的持续视觉语言学习应用
对比语言-图像预训练（CLIP）模型在理解图文关系方面表现优异，但在新数据上适应时易遗忘旧知识。通常通过微调新任务数据与过去任务的记忆缓冲区来解决，但当记忆缓冲区较小时，CLIP的对比损失会下降，导致旧任务性能退化。本文提出一种内存高效、分布鲁棒的方法，训练过程中动态重加权每类损失。该方法在类别增量设置（CIFAR-100、ImageNet1K）和领域增量设置（DomainNet）下测试，可在极小内存使用下快速适配新任务，同时显著减少灾难性遗忘。
原文摘要 · Abstract (English)
Contrastive Language-Image Pretraining (CLIP) models excel at understanding image-text relationships but struggle with adapting to new data without forgetting prior knowledge. To address this, models are typically fine-tuned using both new task data and a memory buffer of past tasks. However, CLIP's contrastive loss suffers when the memory buffer is small, leading to performance degradation on previous tasks. We propose a memory-efficient, distributionally robust method that dynamically reweights losses per class during training. Our approach, tested on class incremental settings (CIFAR-100, ImageNet1K) and a domain incremental setting (DomainNet) adapts CLIP models quickly while minimizing catastrophic forgetting, even with minimal memory usage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。