EDoRA用分解权重实现高效微调，参数少30倍仍保持高性能。
EDoRA: Efficient Weight-Decomposed Low-Rank Adaptation via Singular Value Decomposition
- 将预训练权重分解为大小与方向分量，冻结低秩矩阵
- 仅需原模型30分之一的可训练参数，性能媲美甚至超过现有方法
- 适合资源受限场景下大模型快速适配各类任务
参数高效微调方法（如LoRA）虽减少可训练参数，但常面临扩展性差和学习模式与全量微调不一致的问题。为此，我们提出高效权重分解低秩微调（EDoRA）：将预训练权重分解为幅度与方向分量，通过奇异值分解初始化低秩矩阵，并在两者间引入小规模可训练矩阵。实验表明，在GLUE基准上，EDoRA相比当前最优方法（如LoRA、DoRA），可实现高达30倍的可训练参数减少，同时保持或超越其性能，是内存受限环境下大模型高效适配的理想方案。代码已开源：https://github.com/Hamid-Nasiri/EDoRA。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning methods, such as LoRA, reduces the number of trainable parameters. However, they often suffer from scalability issues and differences between their learning pattern and full fine-tuning. To overcome these limitations, we propose Efficient Weight-Decomposed Low-Rank Adaptation (EDoRA): a novel PEFT method that decomposes pre-trained weights into magnitude and directional components. By freezing low-rank matrices, initializing them by singular value decomposition, and introducing a small trainable matrix between them, EDoRA achieves substantial reduction in trainable parameters while maintaining learning capacity. Experimental results on the GLUE benchmark demonstrate that EDoRA achieves competitive or superior performance compared to state-of-the-art methods, such as LoRA and DoRA, with up to 30x fewer trainable parameters. This makes EDoRA a highly efficient solution for adapting LLMs to diverse tasks under memory-constrained settings. Code is available at https://github.com/Hamid-Nasiri/EDoRA .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。