提出CORA方法,用共享正交旋转提升低秩微调的稳定性与效率。
CORA: Per-Slice Coherent Orthogonal Rotation for SVD-based Low-Rank Adaptation

- 每层使用共享正交旋转和对角缩放,保持奇异基耦合几何
- 参数量仅为LoRA的1/4,代码生成与常识推理更优
- 适合追求高效微调的模型部署场景
参数高效微调(PEFT)通常通过低秩更新适配预训练权重,近期方法利用基权重的奇异值分解(SVD)进行初始化或子空间选择。然而,这些方法未显式保留预训练左右奇异基之间的耦合几何。受最小扰动理论启发,该理论表明稳定微调需在左右奇异基上施加同一正交变换$Q$,我们证明了逐切片的等价形式:基权重$W_0$的每一行切片可由共享正交旋转$Q_i$同时作用于其左右奇异基$U_i$和$V_i$,并配合对角谱移。我们实现该形式为CORA(Coherent Orthogonal Rotation Adaptation),在$W_0$的秩$r$ SVD截断上应用逐切片正交旋转和逐层对角缩放。CORA每线性层仅需$\tfrac{1}{2}m(r{-}1)$个可训练参数,约是同秩下LoRA的1/4。CORA在常识推理与代码生成任务上优于LoRA、DoRA、PiSSA和MiLoRA,且参数量约为其1/8。
原文摘要 · Abstract (English)
Parameter-Efficient Fine-Tuning (PEFT) commonly adapts pretrained weights through low-rank updates, and recent methods further exploit the singular value decomposition (SVD) of the base weight for initialization or subspace selection. However, these methods do not explicitly preserve the coupled geometry between the pretrained left and right singular bases. Motivated by recent minimum-perturbation theory, which shows that stable finetuning follows a coherent SVD rotation in which a single orthogonal $Q$ acts on both the left singular basis $U_0$ and the right singular basis $V_0$, we prove a per-slice analogue: each row slice of $W_0$ can be adapted by a shared orthogonal rotation $Q_i$ on its left basis $U_i$ and right basis $V_i$ together with a diagonal spectrum shift. We implement this form as CORA (Coherent Orthogonal Rotation Adaptation), which applies per-slice orthogonal rotations and a per-layer diagonal scale to the rank-$r$ SVD truncation of $W_0$. CORA uses $\tfrac{1}{2}m(r{-}1)$ trainable parameters per linear layer, about $4{\times}$ fewer than LoRA at the same rank. CORA outperforms LoRA, DoRA, PiSSA, and MiLoRA on commonsense reasoning and code generation while using about $8{\times}$ fewer parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。