通过条件数筛选关键低秩矩阵,实现更高效的模型微调。
\k{appa}-LoRA: Condition Numbers Reveal Which LoRA Matrices Worth Updating
- 根据矩阵条件数选择性更新,只优化最具影响力的低秩参数。
- 仅更新前50%条件数最大的矩阵,参数量减半,计算开销降低一半。
- 适合资源受限场景,如边缘设备和移动端微调,兼顾效率与性能。
低秩适配(LoRA)是高效微调神经网络的常用方法,将模型更新分解为低秩矩阵。然而,传统方法对所有矩阵无差别更新,导致计算成本高昂,尤其在参数量达数十亿的大模型及资源受限的边缘部署场景中尤为明显。本文首次发现:并非所有LoRA矩阵都同等重要——条件数较小的矩阵已方向均衡,贡献有限;而条件数较大的矩阵包含未充分发展的方向,能覆盖更丰富的子空间,驱动主要性能提升。基于此,我们提出\k{appa}-LoRA,聚焦于条件数最大的矩阵进行更新。通过仅对前50%条件数最高的矩阵进行训练,\k{appa}-LoRA将可训练参数量减少一半,相应降低计算与内存开销。跨多个基准的实验表明,该方法平均缩短微调时间16.2%,精度与标准LoRA相当,内存消耗降低4.5%。进一步分析显示,被选中的矩阵条件数在训练过程中持续下降,说明其有效性源于针对性的谱重平衡,而非单纯参数选择。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) has become a widely adopted technique for efficient neural network fine-tuning, decomposing model updates into low-rank matrices. However, LoRA remains computationally costly because it updates all matrices uniformly, regardless of their actual contribution to adaptation. This cost is especially prohibitive for large-scale models with billions of parameters and for resource-constrained settings such as edge deployment and on-device fine-tuning. We show for the first time that not all LoRA matrices are equally worth tuning: matrices with smaller condition numbers (the ratio of largest to smallest singular value) are already well-balanced across directions and contribute only marginally to adaptation, whereas matrices with larger condition numbers contain underdeveloped directions that span richer subspaces and drive most of the performance gains. This observation itself is a key contribution of our work, and it motivates a more selective approach to fine-tuning. Building on this insight, we propose \k{appa}-LoRA, a method that optimizes LoRA by focusing updates on the matrices with the largest condition numbers, which capture the most informative directions of change. By restricting LoRA updates to the top 50% of weight matrices ranked by condition number, \k{appa}-LoRA halves the trainable parameter count and correspondingly reduces compute and memory cost. Extensive experiments across multiple benchmarks show that this design cuts fine-tuning time by 16.2% on average while matching the accuracy of standard LoRA and reducing memory cost by 4.5%. Further analysis reveals that the condition numbers of the selected matrices consistently decrease over training, suggesting that \k{appa}-LoRA's effectiveness stems from targeted spectral rebalancing rather than parameter selection alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。