让零样本视觉语言模型无需记忆就能持续学习,避免遗忘。
Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models
- 用近似零空间约束更新参数,减少对旧知识干扰。
- 在不丢零样本能力的前提下,实现接近顶尖的持续学习表现。
- 轻量无记忆设计,适合资源受限的真实场景使用。
预训练的视觉语言模型(如 CLIP)具备出色的零样本泛化能力,可在无需额外任务训练的情况下部署于多种实际任务中。然而,在环境动态变化或出现新类别时,这些模型会遭遇分布偏移和新任务挑战,静态的零样本能力已不足以应对。为此,亟需一种可随时间适应、同时避免灾难性遗忘的持续学习方法。本文提出 NuSA-CL(Null Space Adaptation for Continual Learning),一种轻量级、无记忆的持续学习框架。该方法采用低秩适配,并将任务特定权重更新限制在模型当前参数的近似零空间内,从而最小化对已有知识的干扰,有效保留原始模型的零样本能力。与依赖重放缓冲区或昂贵蒸馏的方法不同,NuSA-CL 计算与内存开销极小,适用于资源受限的真实持续学习场景。实验表明,该框架不仅能有效保持零样本迁移能力,还在多个持续学习基准上取得具有竞争力的性能。这使其成为真实应用中持续演进的零样本视觉语言模型的实用且可扩展解决方案。
原文摘要 · Abstract (English)
Pre-trained vision-language models (VLMs), such as CLIP, have demonstrated remarkable zero-shot generalization, enabling deployment in a wide range of real-world tasks without additional task-specific training. However, in real deployment scenarios with evolving environments or emerging classes, these models inevitably face distributional shifts and novel tasks. In such contexts, static zero-shot capabilities are insufficient, and there is a growing need for continual learning methods that allow models to adapt over time while avoiding catastrophic forgetting. We introduce NuSA-CL (Null Space Adaptation for Continual Learning), a lightweight memory-free continual learning framework designed to address this challenge. NuSA-CL employs low-rank adaptation and constrains task-specific weight updates to lie within an approximate null space of the model's current parameters. This strategy minimizes interference with previously acquired knowledge, effectively preserving the zero-shot capabilities of the original model. Unlike methods relying on replay buffers or costly distillation, NuSA-CL imposes minimal computational and memory overhead, making it practical for deployment in resource-constrained, real-world continual learning environments. Experiments show that our framework not only effectively preserves zero-shot transfer capabilities but also achieves highly competitive performance on continual learning benchmarks. These results position NuSA-CL as a practical and scalable solution for continually evolving zero-shot VLMs in real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。