提出SiGMA框架，缓解多模态持续学习中的知识干扰问题。
SiGMA: Sign-Guided Merging and Adaptation for Multimodal Continual Instruction Tuning
- 通过符号引导的自适应调优减少新旧知识冲突。
- 推理时符号引导合并，选择性增强关键参数以保留有用知识。
- 在UCIT和DCL上优于现有方法，适合持续学习场景。
多模态持续指令微调（MCIT）对适应多模态大模型应对不断变化的下游任务至关重要。以往方法多采用专家混合或扩展融合策略，主要关注灾难性遗忘，但仍存在推理阶段的负向干扰——新知识覆盖已有有效知识，导致整体性能下降。为此，我们提出SiGMA（符号引导的合并与适配），一个简单但有效的框架，包含两个组件：训练时的符号引导自适应调优和推理时的符号引导合并。前者减少与历史知识的冲突，以最小漂移学习当前任务，缓解严重遗忘；后者通过选择性缩放显著参数，强化并保留有用的特定任务知识。在UCIT和DCL基准上的实验表明，SiGMA显著降低负向干扰，优于现有最先进MCIT方法。代码已开源。
原文摘要 · Abstract (English)
Multimodal Continual Instruction Tuning (MCIT) is crucial for adapting Multimodal Large Language Models (MLLMs) to evolving a sequence of downstream tasks. Prior methods mostly utilize Mixture of Experts or expansion merge approach, primarily focusing on catastrophic forgetting, yet they still suffer from negative interference during inference, where newly learned updates overwrite useful prior knowledge and degrade overall performance. To address this, we propose SiGMA (Sign Guided Merging and Adaptation), a simple yet effective framework that mitigates negative interference with two components: sign guided adaptive tuning during training and sign guided merging at inference. Sign guided adaptive tuning reduces collisions with past knowledge and learns the current task with minimal drift, mitigating severe forgetting. Sign guided merging further improves consolidation by selectively scaling salient parameters to preserve and amplify useful task specific knowledge. Experiments on UCIT and DCL benchmarks show that SiGMA significantly reduces negative interference and outperforms state of the art MCIT methods. Our code is available at SiGMA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。