提出GNSP方法，防止视觉语言模型持续学习时遗忘跨模态对齐能力。
GNSP: Gradient Null Space Projection for Preserving Cross-Modal Alignment in VLMs Continual Learning
- 通过梯度投影到已学知识的零空间，避免干扰旧任务。
- 在MTIL基准上达到最优平均与最后任务性能。
- 无需重放或修改结构，适合长期更新的多模态模型。
对比语言-图像预训练（CLIP）通过将视觉与文本模态对齐于共享嵌入空间，展现出出色的零样本泛化能力。然而，在持续微调多个任务时，CLIP会因灾难性遗忘导致嵌入对齐退化，削弱其零样本性能。本文提出梯度零空间投影（GNSP），将任务特定梯度投影至先前知识的零空间中，实现数学上不干扰旧任务的正交投影，无需重放或架构修改。为保持CLIP固有的泛化能力，引入知识蒸馏，并结合受CLIP预训练启发的模态对齐保持损失，稳定微调过程中多模态嵌入空间结构。在包含11个任务的MTIL基准上，本方法在平均与最后任务指标上均达最佳表现。实验表明，该方法成功维持了CLIP原有的模态间隔与跨模态检索性能，验证了其在持续学习过程中保持鲁棒视觉-语言空间的有效性。
原文摘要 · Abstract (English)
Contrastive Language-Image Pretraining has demonstrated remarkable zero-shot generalization by aligning visual and textual modalities in a shared embedding space. However, when continuously fine-tuned on diverse tasks, CLIP suffers from catastrophic forgetting and degradation of its embedding alignment, undermining its zero-shot capabilities. In this work, we propose Gradient Null Space Projection (GNSP), an efficient continual learning method that projects task-specific gradients onto the null space of previously learned knowledge. This orthogonal projection mathematically prevents interference with previous tasks without relying on rehearsal or architectural modification. Furthermore, to preserve the inherent generalization property of CLIP, we introduce knowledge distillation and combine it with a modality alignment preservation loss inspired by CLIP pre-training to stabilize the structure of the multimodal embedding space during fine-tuning. On the MTIL benchmark consisting of 11 tasks, our method achieved SOTA performance on both the Average and Last key metrics. More importantly, experiments show that our method successfully maintains the original modality gap and cross-modal retrieval performance of CLIP, confirming its effectiveness in maintaining a robust visual-language space throughout the continual learning process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。