通过归一化降维矩阵，提升LoRA的训练稳定性与收敛速度。
Normalized Low-Rank Adaptation
- 归一化低秩适配中的下投影矩阵，稳定早期优化过程。
- 在预训练、微调和强化学习中均加速收敛并减少灾难性遗忘。
- 无需额外参数或推理开销，可直接替代标准LoRA。
尽管低秩适配（LoRA）被广泛用于参数高效模型微调，但如何对其训练动态进行正则化以实现稳定高效的优化仍缺乏研究。由于LoRA将上投影初始化为零，其早期优化主要受下投影影响。基于此观察，我们提出归一化低秩适配（NoRA），一种简单有效的训练方法：在训练过程中对下投影矩阵进行归一化。进一步表明，仅在初始化时应用相同归一化即可改善标准LoRA，无需全程重复归一化。在预训练、监督微调和强化学习任务中，NoRA始终加速收敛、提升性能并增强训练稳定性，同时缓解灾难性遗忘。这些优势不依赖额外可训练参数或推理时计算开销，使NoRA成为对LoRA的简单且普适的改进。
原文摘要 · Abstract (English)
While low-rank adaptation (LoRA) is widely used for parameter-efficient model adaptation, how to regularize its training dynamics for stable and effective optimization remains underexplored. Because LoRA initializes the up-projection to zero, its early optimization dynamics are largely governed by the down-projection. Building on this observation, we introduce Normalized Low-Rank Adaptation (NoRA), a simple yet effective method that normalizes the down-projection matrices during training. We further show that the same normalization can be applied only at initialization, improving standard LoRA without requiring repeated normalization throughout training. Across pretraining, supervised finetuning, and reinforcement learning, NoRA consistently accelerates convergence, improves performance and training stability, and mitigates catastrophic forgetting. These benefits require neither additional trainable parameters nor inference-time computation, making NoRA a simple and broadly applicable enhancement to LoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。