用几何约束优化LoRA，提升大模型微调效率
Riemannian Optimization for LoRA on the Stiefel Manifold
- 将LoRA的B矩阵优化置于Stiefel流形上，强制正交性
- 实现近乎完美的正交性和完整有效秩，性能显著提升
- 适合追求高效微调的大模型研究者和工程师
大型语言模型（LLMs）虽强大，但因规模庞大导致微调困难。参数高效微调（PEFT）方法如LoRA提供了解决方案，却面临优化器效率低下的问题，尤其在使用AdamW时，LoRA的B矩阵存在基础向量冗余，根本上限制了性能。本文通过在Stiefel流形上优化B矩阵，施加显式正交性约束，实现了近乎完美的正交性和完整有效秩，显著提升参数效率与表征能力。所提出的Stiefel优化器在多个基准测试中持续优于AdamW，无论采用LoRA或DoRA，均证明几何约束是释放LoRA全部潜力的关键。
原文摘要 · Abstract (English)
While powerful, large language models (LLMs) present significant fine-tuning challenges due to their size. Parameter-efficient fine-tuning (PEFT) methods like LoRA provide solutions, yet suffer from critical optimizer inefficiencies; notably basis redundancy in LoRA's $B$ matrix when using AdamW, which fundamentally limits performance. We address this by optimizing the $B$ matrix on the Stiefel manifold, imposing explicit orthogonality constraints that achieve near-perfect orthogonality and full effective rank. This geometric approach dramatically enhances parameter efficiency and representational capacity. Our Stiefel optimizer consistently outperforms AdamW across benchmarks with both LoRA and DoRA, demonstrating that geometric constraints are the key to unlocking LoRA's full potential for effective LLM fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。