提出动态保留机制，让多模态大模型微调时不丢失已有知识。
AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention
- 根据激活强度自动评估参数重要性，选择性冻结关键参数。
- 在多个下游任务上实现性能超越现有方法，同时保持上游能力。
- 无需修改模型结构，兼容现有推理引擎，适合实际部署。
多模态大语言模型（MLLMs）因大规模多模态预训练展现出强大的泛化与推理能力。然而，在下游任务微调时常出现灾难性遗忘，新学知识会损害原有能力。这是由于新任务的梯度更新覆盖了对先前知识至关重要的参数。为此，我们提出激活加权自适应保留（AWARe）方法，通过基于激活模式动态控制参数更新，缓解灾难性遗忘。AWARe为参数分配基于激活的重要度评分，选择性冻结对保留先验能力至关重要的参数，同时允许不重要参数适应新任务。重要的是，AWARe无需修改模型架构，确保与现有推理引擎兼容。大量实验表明，相比现有方法，AWARe在保持上游能力的同时实现了更优的下游性能。代码已开源：https://github.com/kaln27/AWARe。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) exhibit strong generalization and reasoning abilities due to large-scale multimodal pre-training. However, fine-tuning these models on downstream tasks often leads to catastrophic forgetting, where newly learned task-specific knowledge degrades previously acquired capabilities. This issue arises because gradient updates for new tasks overwrite parameters critical to prior knowledge, limiting the practical deployment of MLLMs. To address this challenge, we propose Activation-Weighted Adaptive REtention (AWARe), a fine-tuning method that mitigates catastrophic forgetting by dynamically controlling parameter updates based on activation patterns. AWARe assigns activation-based importance scores to parameters, selectively freezing those essential for preserving prior capabilities while allowing less important parameters to adapt to new tasks. Importantly, AWARe operates without modifying model architectures, ensuring compatibility with existing inference engines. Extensive experiments demonstrate that AWARe effectively preserves upstream capabilities while achieving superior downstream performance compared to existing methods. Code is available at https://github.com/kaln27/AWARe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。