解决多模态模型持续学习中的感知与推理双重遗忘问题。
MAny: Merge Anything for Multimodal Continual Instruction Tuning

- 通过跨模态投影合并与低秩参数合并,实现任务知识高效融合。
- 在UCIT基准上比顶尖方法最高提升8.57%准确率,显著缓解遗忘。
- 无需训练,仅用CPU算术操作即可完成知识合并,适合快速部署。
多模态持续指令微调(MCIT)对多模态大语言模型(MLLMs)的连续任务适应至关重要,但受灾难性遗忘严重制约。现有研究多关注语言主干推理能力,本文揭示了跨模态投影空间中的感知漂移与低秩参数空间中的推理崩溃这一被忽视的双重遗忘现象。为此,我们提出 extbf{MAny}( extbf{M}erge extbf{Any}thing),通过 extbf{C}ross-modal extbf{P}rojection extbf{M}erging(CPM)与 extbf{L}ow-rank extbf{P}arameter extbf{M}erging(LPM)实现任务知识融合。CPM 利用视觉原型引导,自适应合并跨模态视觉表示,恢复感知对齐;LPM 通过递归合并低秩权重矩阵消除任务间干扰,基于递归最小二乘法提供闭式解,数学保证推理稳定性。MAny 为无训练范式,仅通过高效CPU代数运算完成知识融合,无需额外梯度优化。大量实验验证其在多个MLLM与基准上的优越性能:在UCIT基准上,相比最先进方法,两个不同MLLM分别取得最高达8.57%和2.85%的最终平均准确率提升。
原文摘要 · Abstract (English)
Multimodal Continual Instruction Tuning (MCIT) is essential for sequential task adaptation of Multimodal Large Language Models (MLLMs) but is severely restricted by catastrophic forgetting. While existing literature focuses on the reasoning language backbone, in this work, we expose a critical yet neglected dual-forgetting phenomenon across both perception drift in Cross-modal Projection Space and reasoning collapse in Low-rank Parameter Space. To resolve this, we present \textbf{MAny} (\textbf{M}erge \textbf{Any}thing), a framework that merges task-specific knowledge through \textbf{C}ross-modal \textbf{P}rojection \textbf{M}erging (\textbf{CPM}) and \textbf{L}ow-rank \textbf{P}arameter \textbf{M}erging (\textbf{LPM}). Specifically, CPM recovers perceptual alignment by adaptively merging cross-modal visual representations via visual-prototype guidance, ensuring accurate feature recovery during inference. Simultaneously, LPM eliminates mutual interference among task-specific low-rank modules by recursively merging low-rank weight matrices. By leveraging recursive least squares, LPM provides a closed-form solution that mathematically guarantees an optimal fusion trajectory for reasoning stability. Notably, MAny operates as a training-free paradigm that achieves knowledge merging via efficient CPU-based algebraic operations, eliminating additional gradient-based optimization beyond initial tuning. Our extensive evaluations confirm the superior performance and robustness of MAny across multiple MLLMs and benchmarks. Specifically, on the UCIT benchmark, MAny achieves significant leads of up to 8.57\% and 2.85\% in final average accuracy over state-of-the-art methods across two different MLLMs, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。