用稀疏更新和正则化让LoRA同时保留通用与专用知识
LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models
- 在LoRA中引入稀疏更新，剔除冗余参数
- 5%以下稀疏度下仍提升通用与下游任务性能
- 适合需要平衡通用性与定制化的多模态模型优化
尽管多模态大语言模型（MLLMs）在跨模态和任务间具有强泛化能力，但在适配特定下游任务时，如何同时保持通用知识与专用知识仍具挑战。尽管低秩适配（LoRA）被广泛用于高效获取专用知识，但其在视觉指令微调中引入了大量有害冗余参数，加剧了通用知识遗忘，降低下游性能。为此，我们提出LoRASculpt，通过理论保证的稀疏更新有效剔除冗余参数，实现通用与专用知识的协调统一。此外，设计冲突缓解正则化器，优化LoRA更新轨迹，缓解与预训练权重的知识冲突。大量实验表明，即使在极低稀疏度（≤5%）下，该方法仍能同时提升泛化能力与下游任务表现，证实其有效缓解灾难性遗忘，并促进MLLM中的知识协同。
原文摘要 · Abstract (English)
While Multimodal Large Language Models (MLLMs) excel at generalizing across modalities and tasks, effectively adapting them to specific downstream tasks while simultaneously retaining both general and specialized knowledge remains challenging. Although Low-Rank Adaptation (LoRA) is widely used to efficiently acquire specialized knowledge in MLLMs, it introduces substantial harmful redundancy during visual instruction tuning, which exacerbates the forgetting of general knowledge and degrades downstream task performance. To address this issue, we propose LoRASculpt to eliminate harmful redundant parameters, thereby harmonizing general and specialized knowledge. Specifically, under theoretical guarantees, we introduce sparse updates into LoRA to discard redundant parameters effectively. Furthermore, we propose a Conflict Mitigation Regularizer to refine the update trajectory of LoRA, mitigating knowledge conflicts with the pretrained weights. Extensive experimental results demonstrate that even at very high degree of sparsity ($\le$ 5%), our method simultaneously enhances generalization and downstream task performance. This confirms that our approach effectively mitigates the catastrophic forgetting issue and further promotes knowledge harmonization in MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。