通过渐进式累积低秩更新，实现高效高秩微调。
ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
- 逐次优化低秩矩阵的缩放系数，动态逼近高秩更新效果。
- 在120亿参数模型上，任务性能超越现有LoRA方法，收敛更快。
- 适合需要高精度微调且资源受限的LLM应用者。
随着大语言模型规模持续扩大，特定任务微调带来的计算开销已成为主要瓶颈。尽管低秩适配（LoRA）通过将权重更新限制在低维子空间有效降低开销，但该限制可能影响性能并减缓收敛速度。本文提出一种渐进式累积高秩更新的方法，通过每次迭代中识别最小化损失函数的最优低秩矩阵，逼近全量微调效果。最优缩放可通过解析方式确定，使优化过程无需重启即可高效进行。在多达120亿参数的主流大模型上，针对自然语言理解、常识推理和数学问题求解等任务的大量实验表明，该方法在性能与收敛速度上均优于当前先进LoRA变体。
原文摘要 · Abstract (English)
As large language models (LLMs) continue to scale in size, the computational overhead has become a major bottleneck for task-specific fine-tuning. While low-rank adaptation (LoRA) effectively curtails this cost by confining the weight updates to a low-dimensional subspace, such a restriction can hinder effectiveness and slow convergence. This contribution deals with these limitations by accumulating progressively a high-rank weight update from consecutive low-rank increments. Specifically, the per update optimal low-rank matrix is identified to minimize the loss function and closely approximate full fine-tuning. To endow efficient and seamless optimization without restarting, this optimal choice is formed by appropriately scaling the columns of the original low-rank matrix. Rigorous performance guarantees reveal that the optimal scaling can be found analytically. Extensive numerical tests with popular LLMs scaling up to 12 billion parameters demonstrate a consistent performance gain and fast convergence relative to state-of-the-art LoRA variants on diverse tasks including natural language understanding, commonsense reasoning, and mathematical problem solving.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。