提出轻量级LoRA变体EffiLoRA，显著降低资源消耗
Less is More: Resource-Efficient Low-Rank Adaptation
- 统一所有层的A矩阵，动态选择性更新B矩阵
- 在多模态任务中性能超越传统LoRA，资源占用更低
- 适合资源受限场景下的大模型微调，通用性强
低秩适配（LoRA）是大语言模型参数高效微调的常用方法，但仍存在明显开销和复杂数据集中的参数干扰问题。尽管近期工作通过解耦更新矩阵以利用矩阵间不对称性，训练成本仍较高。本文从矩阵间与层内参数冗余角度重新审视LoRA，提出一种轻量且通用的高效低秩适配方法EffiLoRA，适用于语言、多模态及扩散模型。EffiLoRA采用统一的A矩阵覆盖所有Transformer层，并引入运行时选择性更新B矩阵机制，动态平衡系统资源预算与模型性能。在常识推理、视觉指令微调与图像生成等多种任务中，EffiLoRA持续优于标准LoRA，展现出更优的效率与鲁棒性。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs), but it still incurs notable overhead and suffers from parameter interference in complex datasets. While recent works decouple LoRA update matrices to exploit matrix-wise asymmetry, training costs remain high. We revisit LoRA from the perspective of inter-matrix and intra-layer parameter redundancy and propose Resource-Efficient Low-Rank Adaptation, EffiLoRA, a lightweight and generalizable approach for language, multimodal, and diffusion models. EffiLoRA employs a unified A matrix across all transformer layers and introduces a runtime selective B matrices update to dynamically trade-off the system resource budget and model performance. EffiLoRA consistently outperforms LoRA across diverse modalities, including commonsense reasoning, visual instruction tuning, and image generation, demonstrating improved efficiency and robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。