arXiv:2505.12433cs.CVcs.AI2025-05被引 2

SRLoRA让低秩微调更灵活,动态重组更新空间提升性能。

SRLoRA: Subspace Recomposition in Low-Rank Adaptation via Importance-Based Fusion and Reinitialization

  • 根据重要性动态融合低秩对,重用预训练模型主方向
  • 训练中持续刷新子空间,准确率高于标准LoRA
  • 适合追求高效微调且想提升表达能力的研究者

低秩适配(LoRA)是一种广泛使用的参数高效微调方法,通过在冻结的预训练模型中注入两个可训练的低秩矩阵(A和B)实现。尽管高效,但LoRA将更新限制在固定的低秩子空间(ΔW = BA),可能限制表达能力并影响下游性能。我们提出基于重要性融合与重初始化的低秩适配子空间重构(SRLoRA),一种增强LoRA表达力的新方法,同时保持轻量结构。SRLoRA为每个LoRA对(B的列与对应A的行)分配重要性评分,并在训练过程中动态重组子空间。重要性较低的对被融合进冻结主干网络,释放容量用于沿预训练权重奇异值分解所得未使用主方向重新初始化新对。该机制实现持续子空间刷新,随时间提升适应能力,且不增加可训练参数数量。我们在语言与视觉任务上评估SRLoRA,包括GLUE基准和多个图像分类数据集。结果表明,相比标准LoRA,SRLoRA在收敛速度和准确率上均表现更优,展现其通用性、高效性及在更广参数高效微调场景中的潜力。

原文摘要 · Abstract (English)

Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning (PEFT) method that injects two trainable low-rank matrices (A and B) into frozen pretrained models. While efficient, LoRA constrains updates to a fixed low-rank subspace (Delta W = BA), which can limit representational capacity and hinder downstream performance. We introduce Subspace Recomposition in Low-Rank Adaptation (SRLoRA) via importance-based fusion and reinitialization, a novel approach that enhances LoRA's expressiveness without compromising its lightweight structure. SRLoRA assigns importance scores to each LoRA pair (a column of B and the corresponding row of A), and dynamically recomposes the subspace during training. Less important pairs are fused into the frozen backbone, freeing capacity to reinitialize new pairs along unused principal directions derived from the pretrained weight's singular value decomposition. This mechanism enables continual subspace refreshment and richer adaptation over time, without increasing the number of trainable parameters. We evaluate SRLoRA on both language and vision tasks, including the GLUE benchmark and various image classification datasets. SRLoRA consistently achieves faster convergence and improved accuracy over standard LoRA, demonstrating its generality, efficiency, and potential for broader PEFT applications.

低秩微调参数效率模型优化子空间重构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。