不依赖数据也能评估参数重要性，缓解多模态大模型微调时的记忆丢失。
Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Models
- 通过权重、激活与敏感度综合评估参数重要性。
- 在LLaVA和NVILA上显著减少遗忘，保持性能稳定。
- 无需额外数据，适合超大规模模型高效微调。
在特定任务数据上微调多模态大语言模型（MLLMs）能提升下游性能，但常导致预训练任务泛化能力下降，即灾难性遗忘。现有方法在微调语言解码器深层时失效，或随模型规模增长而效率骤降。为此，我们提出Model-Dowser，一种新型稀疏微调方法。该方法通过联合考虑权重大小、输入激活和输出敏感度，为每个参数计算针对预训练泛化能力的合理重要性得分。微调时，仅保留高重要性参数，其余参数更新。在两个代表性模型LLaVA和NVILA上的实验表明，Model-Dowser有效缓解灾难性遗忘，持续优于已有方法，且资源消耗低、可扩展至数十亿参数模型。
原文摘要 · Abstract (English)
Fine-tuning Multimodal Large Language Models (MLLMs) on task-specific data is an effective way to improve performance on downstream applications. However, such adaptation often leads to a degradation in generalization on pretrained tasks, a phenomenon known as Catastrophic Forgetting. Existing methods that aim to mitigate this issue either become ineffective when fine-tuning deeper layers of the language decoder or scale poorly with increasing model size. To address these limitations, we propose Model-Dowser, a novel sparse fine-tuning approach for MLLMs. Model-Dowser measures a principled importance score for each model parameter with respect to pretrained generalization (prior to downstream adaptation) by jointly considering weight magnitudes, input activations, and output sensitivities. During fine-tuning, Model-Dowser selectively preserves high-importance parameters and updates the remaining. Comprehensive experiments on two representative MLLMs, LLaVA and NVILA, demonstrate that Model-Dowser effectively mitigates catastrophic forgetting and consistently outperforms prior methods, while remaining resource-efficient and scalable to multi-billion-parameter models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。