用尼尔斯滕方法加速大模型微调，性能提升且参数极少。
NLoRA: Nyström-Initiated Low-Rank Adaptation for Large Language Models
- 引入中间矩阵结构，改进低秩微调的初始化方式。
- 在GSM8K上准确率超LoRA 36.41%，仅增367万参数。
- 只需0.0125倍参数量，就实现比传统方法高7.45%效果。
参数高效微调（PEFT）对大语言模型（LLM）至关重要，其中低秩适配（LoRA）最为流行。然而，LoRA收敛缓慢，部分新变体如PiSSA依赖奇异值分解（SVD）初始化，计算成本高。为此，我们采用尼尔斯滕方法，通过三矩阵操作实现高效初始化。首先提出结构化低秩适配（SLoRA），在低秩矩阵A与B之间加入小型中间矩阵；其次提出尼尔斯滕低秩适配（NLoRA），基于尼尔斯滕初始化增强SLoRA的效率与效果；最后提出中间微调（IntTune），仅微调NLoRA的中间矩阵以进一步提升效率。我们在五个自然语言生成（NLG）任务和八个自然语言理解（NLU）任务上评估，结果显示，在GSM8K上，SLoRA与NLoRA准确率分别为56.48%和57.70%，较LoRA分别提升33.52%和36.41%，额外参数仅367万。IntTune相较LoRA平均提升NLG性能7.45%，参数占比仅为1.25%。结果表明，该方法在极小参数开销下显著提升模型表现。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) is essential for adapting large language models (LLMs), with low-rank adaptation (LoRA) being the most popular approach. However, LoRA suffers from slow convergence, and some recent LoRA variants, such as PiSSA, primarily rely on Singular Value Decomposition (SVD) for initialization, leading to expensive computation. To mitigate these problems, we use the Nyström method, which follows a three-matrix manipulation. We first introduce StructuredLoRA (SLoRA), which investigates adding a small intermediate matrix between the low-rank matrices A and B. Secondly, we propose NyströmLoRA (NLoRA), which leverages Nyström-based initialization for SLoRA to improve its effectiveness and efficiency. Finally, we propose IntermediateTune (IntTune), which explores fine-tuning exclusively on the intermediate matrix of NLoRA to further boost LLM efficiency. We evaluate our methods on five natural language generation (NLG) tasks and eight natural language understanding (NLU) tasks. On GSM8K, SLoRA and NLoRA achieve accuracies of 56.48% and 57.70%, surpassing LoRA by 33.52% and 36.41%, with only 3.67 million additional trainable parameters. IntTune improves average NLG performance over LoRA by 7.45% while using only 1.25% of its parameters. These results demonstrate the efficiency and effectiveness of our approach in enhancing model performance with minimal parameter overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。