通过局部增强适配器，高效解决多模态模型微调中的数据冲突问题。
From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning
- 分设技能空间与修正秩任务空间，实现全局知识保留与局部激活。
- 仅需标准LoRA 1.16倍推理时间，比4专家LoRA-MoE快27%。
- 适合追求高效微调且关注视觉细节的多模态研究者。
高效视觉指令微调（EVIT）旨在以最小计算开销将多模态大语言模型（MLLMs）适配至下游任务。然而，随着任务多样性与复杂性提升，EVIT面临显著的数据冲突挑战。为此，我们提出双低秩适配（Dual-LoRA），一种从全局到局部的框架，通过双重结构优化增强适配器应对数据冲突的能力。具体地，引入两个子空间：用于稳定保留全局知识的技能空间，以及局部激活全局知识的秩修正任务空间。此外，提出视觉线索增强（VCE）模块，一种多层次局部特征聚合机制，以丰富视觉-语言投影中的局部细节。该方法在内存与时间上均高效，仅需标准LoRA方法（注入查询与值投影层）1.16倍的推理时间，且仅为4专家LoRA-MoE的73%。在多种下游任务及通用MLLM基准上的广泛实验验证了所提方法的有效性。
原文摘要 · Abstract (English)
Efficient Visual Instruction Fine-Tuning (EVIT) seeks to adapt Multimodal Large Language Models (MLLMs) to downstream tasks with minimal computational overhead. However, as task diversity and complexity increase, EVIT faces significant challenges in resolving data conflicts. To address this limitation, we propose the Dual Low-Rank Adaptation (Dual-LoRA), a holistic-to-local framework that enhances the adapter's capacity to address data conflict through dual structural optimization. Specifically, we utilize two subspaces: a skill space for stable, holistic knowledge retention, and a rank-rectified task space that locally activates the holistic knowledge. Additionally, we introduce Visual Cue Enhancement (VCE), a multi-level local feature aggregation module designed to enrich the vision-language projection with local details. Our approach is both memory- and time-efficient, requiring only 1.16$\times$ the inference time of the standard LoRA method (with injection into the query and value projection layers), and just 73\% of the inference time of a 4-expert LoRA-MoE. Extensive experiments on various downstream tasks and general MLLM benchmarks validate the effectiveness of our proposed methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。