SORSA通过奇异值与正交约束提升大模型微调效率，性能优于LoRA。
SORSA: Singular Values and Orthonormal Regularized Singular Vectors Adaptation of Large Language Models
- 用奇异值分解初始化可训练权重，结合正交约束优化
- 在GSM-8K上达到56.03%准确率，超越LoRA与全量微调
- 适合作为高效微调方法，尤其适合资源受限场景
本文提出一种新型参数高效微调方法SORSA，其适配器由可训练的主奇异权重 $W_p = U_p \text{diag}(S_p) V^ op_p$ 和冻结的残差权重 $W_r = U_r \text{diag}(S_r) V^ op_r$ 构成，均通过预训练权重的奇异值分解（SVD）初始化。我们引入正交正则化项，证明其可降低 $W_p$ 的条件数，提升优化效率。SORSA适配器可在推理时合并，消除延迟。通过SVD分析参数变化，验证其在保持奇异值结构方面更优。实验表明，SORSA收敛更快；在GSM-8K基准上，Llama 2 7B使用SORSA微调达56.03%准确率，优于LoRA（42.30%）和全量微调（49.05%），展现出显著性能优势。
原文摘要 · Abstract (English)
In this paper, we propose Singular Values and Orthonormal Regularized Singular Vectors Adaptation, or SORSA, a novel parameter efficient fine-tuning (PEFT) method. Each SORSA adapter consists of two main parts: trainable principal singular weights $W_p = U_p \text{diag}(S_p) V^\top_p$, and frozen residual weights $W_r = U_r \text{diag}(S_r) V^\top_r$. These parts are initialized by performing singular value decomposition (SVD) on pre-trained weights. Moreover, we implement and analyze an orthonormal regularizer, which we prove could decrease the condition number of $W_p$ and make the optimization more efficient. SORSA adapters could be merged during inference, thus eliminating any inference latency. We also introduce a method to analyze the variation of the parameters by performing SVD and discuss and analyze SORSA's superiority in minimizing the alteration in the SVD aspect. After all, SORSA shows a faster convergence than LoRA and PiSSA in our experiments. On the GSM-8K benchmark, Llama 2 7B adapted using SORSA achieved 56.03\% accuracy, surpassing LoRA (42.30\%) and Full FT (49.05\%). We conclude that SORSA offers a new perspective on parameter-efficient fine-tuning, demonstrating remarkable performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。