改进LoRA训练速度，通过平衡参数提升收敛效率。
Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence
- 提出平衡流形投影机制，解决LoRA的参数冗余问题。
- 在多个任务上收敛速度更快，性能优于标准LoRA。
- 无需修改训练流程，适合现有大模型微调场景。
低秩适配（LoRA）是微调大语言模型最广泛采用的方法。值得注意的是，LoRA本质上是过参数化的：多组低秩因子可生成相同的适配权重矩阵。我们通过理论与实证证明，这些因子对具有显著不同的条件数，从而直接影响LoRA的收敛速率。基于此观察，我们提出平衡低秩适配（BaLoRA），一种将迭代点投影至平衡流形的LoRA变体。该流形改善了损失景观的条件性，同时保持适配矩阵不变。投影步骤计算开销小，可无缝集成至现有微调流程。实验表明，BaLoRA相比标准LoRA收敛更快，且在多种微调任务中表现更优。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models. Notably, LoRA is inherently overparameterized: multiple pairs of low-rank factors can yield the same adapted weight matrix. We show--both theoretically and empirically--that these pairs exhibit significantly different condition numbers. As a result, converging to different loss minimizers directly impacts the convergence rate of LoRA. Building on this observation, we introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold. This manifold improves the conditioning of the loss landscape while preserving the adapted matrix. The projection step is computationally lightweight and integrates seamlessly into existing fine-tuning pipelines. Empirically, BaLoRA converges faster than standard LoRA and achieves superior performance across a range of fine-tuning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。