用模型合并缓解视觉语言模型的持续学习遗忘问题
Continual Learning in Vision-Language Models via Aligned Model Merging
- 通过合并新旧任务参数，平衡模型稳定性与可塑性
- 显著降低遗忘率，提升对任务顺序和相似性的鲁棒性
- 适合需要长期增量学习的视觉语言系统应用
持续学习传统上依赖顺序微调，虽能适应新任务，但易导致先前知识丢失。现有方法虽试图缓解灾难性遗忘，仍因依赖顺序更新而偏向近期任务。本文提出基于模型合并的新视角：不单纯更新权重，而是将新训练参数与已有参数对齐合并，以实现更优的稳定与可塑性平衡。为提升合并效果，设计了促进权重对齐的简单机制，避免合并时的干扰。在大型视觉语言模型上验证表明，该方法有效减少遗忘、增强对任务顺序与相似性的鲁棒性，并提升泛化能力。
原文摘要 · Abstract (English)
Continual learning is conventionally tackled through sequential fine-tuning, a process that, while enabling adaptation, inherently favors plasticity over the stability needed to retain prior knowledge. While existing approaches attempt to mitigate catastrophic forgetting, a bias towards recent tasks persists as they build upon this sequential nature. In this work we present a new perspective based on model merging to maintain stability while still retaining plasticity. Rather than just sequentially updating the model weights, we propose merging newly trained task parameters with previously learned ones, promoting a better balance. To maximize the effectiveness of the merging process, we propose a simple mechanism that promotes learning aligned weights with previous ones, thereby avoiding interference when merging. We evaluate this approach on large Vision-Language Models (VLMs), and demonstrate its effectiveness in reducing forgetting, increasing robustness to various task orders and similarities, and improving generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。