用曲率指导参数融合，解决多模态模型微调后的遗忘问题。
Curvature-Guided Mixing for MLLM Adaptation
- 基于损失曲面的二阶近似，自动计算最优软混合比例。
- 在多个任务上比现有方法提升专用能力与通用知识保留的平衡。
- 适合需要保留模型通用能力的下游微调场景。
在特定任务上微调多模态大语言模型（MLLM）常导致其通用能力严重退化。现有模型融合方法多依赖启发式或次优目标。本文提出曲率引导混合（CGM），一种理论严谨的融合框架，通过联合优化目标和损失曲面的二阶（Hessian）近似，解析推导出最优闭式“软混合”比例，该比例根据参数的任务特定曲率智能融合。我们还引入CGM†，一种鲁棒的“硬混合”变体，通过新颖的曲率感知评分进行稀疏参数选择。在LLaVA-1.5和Qwen2.5VL上的实验表明，CGM与CGM†在多个下游任务中持续优于现有方法，显著改善任务专业化与通用知识保留之间的权衡。代码已公开于github.com/zzsyjl/CGM-ECCV-2026。
原文摘要 · Abstract (English)
Fine-tuning Multimodal Large Language Models (MLLMs) on specialized tasks often leads to catastrophic forgetting of their general capabilities. Existing model merging methods to combat this are often heuristic or use sub-optimal objectives. We propose CurvatureGuided Mixing (CGM), a theoretically grounded framework that merges pre-trained and fine-tuned models. CGM formulates a joint optimization objective and uses a second-order (Hessian) approximation of the loss landscapes to analytically derive an optimal, closed-form "soft mixing" ratio. This ratio intelligently blends parameters based on their relative task-specific curvatures. We also introduce CGM$\dagger$, a robust "hard mixing" variant that performs sparse parameter selection guided by a novel, curvature-aware score. Experiments on LLaVA-1.5 and Qwen2.5VL across multiple downstream tasks show that CGM and CGM$\dagger$ consistently improve the trade-off between task specialization and general knowledge retention over existing methods. Code is available at github.com/zzsyjl/CGM-ECCV-2026.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。