通过定位关键参数并选择性融合，缓解多模态大模型训练中的语言能力退化问题。
Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs
- 先定位重要参数，再按神经元级别选择性合并。
- 在保留视觉能力的同时，显著减少语言性能下降。
- 适合需要保持语言能力的多模态模型微调场景。
尽管多模态大语言模型（MLLMs）取得了优异表现，但在多模态指令微调阶段常导致基础语言模型的语言能力发生灾难性遗忘，即使在强模型如Llama3中也存在此问题。为此，我们提出Locate-then-Merge，一种无需训练的参数融合框架，首先定位关键参数，再进行选择性合并。我们进一步引入Neuron-Fusion，一种神经元级策略，保留参数变化较大的神经元影响——这些神经元可能负责新获得的视觉能力——同时减弱变化较小神经元的影响，这些神经元可能编码通用语言技能。该设计在更好保留视觉适应能力的同时，减轻语言退化。在13个跨语言与视觉任务的基准测试中，Neuron-Fusion持续优于现有模型合并方法。进一步分析显示，该方法有效降低了生成中的上下文幻觉。
原文摘要 · Abstract (English)
Although multimodal large language models (MLLMs) have achieved impressive performance, the multimodal instruction tuning stage often causes catastrophic forgetting of the base LLM's language ability, even in strong models like Llama3. To address this, we propose Locate-then-Merge, a training-free parameter fusion framework that first locates important parameters and then selectively merges them. We further introduce Neuron-Fusion, a neuron-level strategy that preserves the influence of neurons with large parameter shifts--neurons likely responsible for newly acquired visual capabilities--while attenuating the influence of neurons with smaller changes that likely encode general-purpose language skills. This design enables better retention of visual adaptation while mitigating language degradation. Experiments on 13 benchmarks across both language and visual tasks show that Neuron-Fusion consistently outperforms existing model merging methods. Further analysis reveals that our method effectively reduces context hallucination in generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。