让多模态模型同时应对数据冲突与共性，提升适应能力。
AsymLoRA: Harmonizing Data Conflicts and Commonalities in MLLMs
- 用不对称LoRA分离任务特异性与跨模态共性，统一处理数据矛盾与共享信息。
- 在多个基准上性能超越传统LoRA和LoRA-MoE，兼顾效果与效率。
- 适合需要多数据集适配的多模态大模型训练场景。
在多样化图文数据集上进行有效的指令微调对构建通用多模态大模型至关重要，数据集构成决定了模型在多模态任务中的适应能力。然而，复杂数据集常包含源于模态特定优化目标的内在冲突，以及促进跨任务迁移的潜在共性，现有方法通常分别处理。为此，我们提出AsymLoRA，一种参数高效微调框架，通过不对称LoRA统一知识模块化与跨模态协调：任务特异性低秩投影（矩阵B）保留不同目标的独立适应路径，共享投影（矩阵A）整合跨模态共性。大量实验表明，AsymLoRA持续优于仅捕捉共性的原始LoRA和仅关注冲突的LoRA-MoE，跨多种基准实现更优性能与系统效率。
原文摘要 · Abstract (English)
Effective instruction fine-tuning on diverse image-text datasets is crucial for developing a versatile Multimodal Large Language Model (MLLM), where dataset composition dictates the model's adaptability across multimodal tasks. However, complex datasets often contain inherent conflicts -- stemming from modality-specific optimization objectives -- and latent commonalities that enable cross-task transfer, which most existing approaches handle separately. To bridge this gap, we introduce AsymLoRA, a parameter-efficient tuning framework that unifies knowledge modularization and cross-modal coordination via asymmetric LoRA: task-specific low-rank projections (matrix B) that preserve distinct adaptation pathways for conflicting objectives, and a shared projection (matrix A) that consolidates cross-modal commonalities. Extensive evaluations demonstrate that AsymLoRA consistently surpasses both vanilla LoRA, which captures only commonalities, and LoRA-MoE, which focuses solely on conflicts, achieving superior model performance and system efficiency across diverse benchmarks.\href{Code}{https://github.com/Clin0212/HydraLoRA/blob/main/MLLM-HydraLoRA/README.md}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。