arXiv:2506.02041cs.CLcs.AI2025-06ACL被引 17

提出BranchLoRA,解决多模态持续学习中的遗忘问题。

Enhancing Multimodal Continual Instruction Tuning with BranchLoRA

  • 设计非对称分支结构,动态分配任务知识
  • 在多个模型尺寸上超越MoELoRA性能
  • 无需任务标签也能自动路由输入

多模态持续指令微调(MCIT)旨在持续调整多模态大语言模型(MLLMs)以适应人类意图。现有方法依赖混合专家(MoE)LoRA框架保持先前指令对齐,但因简单求和所有LoRA模块而易引发灾难性遗忘。本文揭示了MoELoRA在MCIT中的参数效率低下问题,提出分支型轻量微调框架BranchLoRA。该框架通过灵活的无须微调的冻结机制,使各分支专注任务内知识并促进跨任务协作;逐步引入任务专用路由器,确保分支分布随时间优化,而非偏向最新任务;还设计任务选择器,在推理时自动将输入路由至对应路由器,无需任务标识。在最新MCIT基准上的大量实验表明,BranchLoRA显著优于MoELoRA,并在不同规模的MLLM上均保持优势。

原文摘要 · Abstract (English)

Multimodal Continual Instruction Tuning (MCIT) aims to finetune Multimodal Large Language Models (MLLMs) to continually align with human intent across sequential tasks. Existing approaches often rely on the Mixture-of-Experts (MoE) LoRA framework to preserve previous instruction alignments. However, these methods are prone to Catastrophic Forgetting (CF), as they aggregate all LoRA blocks via simple summation, which compromises performance over time. In this paper, we identify a critical parameter inefficiency in the MoELoRA framework within the MCIT context. Based on this insight, we propose BranchLoRA, an asymmetric framework to enhance both efficiency and performance. To mitigate CF, we introduce a flexible tuning-freezing mechanism within BranchLoRA, enabling branches to specialize in intra-task knowledge while fostering inter-task collaboration. Moreover, we incrementally incorporate task-specific routers to ensure an optimal branch distribution over time, rather than favoring the most recent task. To streamline inference, we introduce a task selector that automatically routes test inputs to the appropriate router without requiring task identity. Extensive experiments on the latest MCIT benchmark demonstrate that BranchLoRA significantly outperforms MoELoRA and maintains its superiority across various MLLM sizes.

多模态持续学习LoRA模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。