LoRA训练几乎总能收敛到低秩全局最优解，失败概率极低。
LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)
- 在真实场景下分析LoRA训练，不依赖线性化假设。
- 训练要么收敛到低秩小幅度全局最优，要么陷入高秩大幅度异常解。
- 权重衰减和零初始化隐式引导参数向低秩最优区域搜索。
低秩适配（LoRA）已成为微调大模型的常用方法。然而，现有对LoRA训练动态的理论分析多依赖线性化假设或简化设定。本文在无此类限制条件下分析了LoRA的损失曲面，区分两种情形：一种是理想化的‘特殊情形’，线性化成立；另一种是更贴近现实的‘通用情形’，线性化不成立。在通用情形下，我们证明LoRA训练会收敛至一个低秩、小幅值的全局最小值，或是一个高秩、大幅值的异类解。最后，我们指出LoRA中的零初始化与权重衰减共同引入隐式偏差，促使优化过程倾向于低秩、小幅值的参数空间——即全局最小值所在区域——从而解释了为何LoRA通常能成功找到全局最优解。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) has become a standard approach for fine-tuning large foundation models. However, our theoretical understanding of LoRA remains limited as prior analyses of LoRA's training dynamics either rely on linearization arguments or consider highly simplified setups. In this work, we analyze the LoRA loss landscape without such restrictive assumptions. We define two regimes: a "special regime", which includes idealized setups where linearization arguments hold, and a "generic regime" representing more realistic setups where linearization arguments do not hold. In the generic regime, we show that LoRA training converges to a global minimizer with low rank and small magnitude, or a qualitatively distinct solution with high rank and large magnitude. Finally, we argue that the zero-initialization and weight decay in LoRA training induce an implicit bias toward the low-rank, small-magnitude region of the parameter space -- where global minima lie -- thus shedding light on why LoRA training usually succeeds in finding global minima.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。