改进矩阵分解初始化，让算法收敛更快更稳定。
On the Crucial Role of Initialization for Matrix Factorization
- 用Nystrom方法初始化，提升梯度下降的收敛速度。
- 在对称和非对称场景下，实现二次收敛，此前仅知线性收敛。
- 适用于10亿到70亿参数模型，显著提升微调效果。
本文重新审视经典的低秩矩阵分解问题，揭示了初始化在非凸、非光滑优化中的关键作用。提出Nystrom初始化方法，显著提升缩放梯度下降（ScaledGD）在对称与非对称矩阵分解中的全局收敛性。具体而言，使用Nystrom初始化的ScaledGD在以往仅知线性收敛的情况下实现了二次收敛。此外，将该初始化方法扩展至常用于微调大模型的低秩适配器（LoRA）。所提方法NoRA（即基于Nystrom初始化的LoRA）在多种下游任务及模型规模（1B至7B参数）的大语言模型与扩散模型中均表现更优。
原文摘要 · Abstract (English)
This work revisits the classical low-rank matrix factorization problem and unveils the critical role of initialization in shaping convergence rates for such nonconvex and nonsmooth optimization. We introduce Nystrom initialization, which significantly improves the global convergence of Scaled Gradient Descent (ScaledGD) in both symmetric and asymmetric matrix factorization tasks. Specifically, we prove that ScaledGD with Nystrom initialization achieves quadratic convergence in cases where only linear rates were previously known. Furthermore, we extend this initialization to low-rank adapters (LoRA) commonly used for finetuning foundation models. Our approach, NoRA, i.e., LoRA with Nystrom initialization, demonstrates superior performance across various downstream tasks and model scales, from 1B to 7B parameters, in large language and diffusion models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。