揭示大模型训练中低秩结构的原理，助力高效微调与训练
An Overview of Low-Rank Structures in the Training and Adaptation of Large Models
- 从梯度下降动态和隐式正则化角度解释低秩现象
- 为LoRA等高效微调方法提供理论支撑
- 适合关注模型压缩与参数效率的研究者
现代大规模深度学习模型带来巨大计算挑战。近期研究发现，深度网络在训练过程中会自然学习到权重与表示中的低秩结构。本文综述了识别与利用这些低秩结构的进展，连接数学基础与实际应用。我们从两个互补视角阐释低秩性的出现：一是训练全程中梯度下降的优化动力学，二是收敛时的隐式正则化效应。这些理论视角为低秩微调（LoRA）的成功提供理解，启发新的参数高效低秩训练策略，并解释了如丢弃法和掩码自监督学习等掩码训练方法的有效性。
原文摘要 · Abstract (English)
The substantial computational demands of modern large-scale deep learning present significant challenges for efficient training and deployment. Recent research has revealed a widespread phenomenon wherein deep networks inherently learn low-rank structures in their weights and representations during training. This tutorial paper provides a comprehensive review of advances in identifying and exploiting these low-rank structures, bridging mathematical foundations with practical applications. We present two complementary theoretical perspectives on the emergence of low-rankness: viewing it through the optimization dynamics of gradient descent throughout training, and understanding it as a result of implicit regularization effects at convergence. Practically, these theoretical perspectives provide a foundation for understanding the success of techniques such as Low-Rank Adaptation (LoRA) in fine-tuning, inspire new parameter-efficient low-rank training strategies, and explain the effectiveness of masked training approaches like dropout and masked self-supervised learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。