对齐低秩适配器改进，权重减半且效果几乎不变
Towards Symmetric Low-Rank Adapters
- 用谱分解代替传统矩阵乘法，实现对称低秩更新
- 参数量仅为传统LoRA的一半，仅损失微量性能
- 适合资源受限场景下的高效微调，如移动端部署
本文提出对称低秩适配器（SymLoRA），一种参数更少的LoRA优化方法。该方法采用低秩对称权重矩阵，通过谱分解形式 $Q \, diag(Λ)\, Q^T$ 学习下游任务，其中 $Q \in \mathbb{R}^{n\times r}$，$Λ\in \mathbb{R}^r$。与传统LoRA使用 $BA$ 形式（$B \in \mathbb{R}^{n\times r}$，$A \in \mathbb{R}^{r\times n}$）不同，新方法将微调权重表示为对称形式，显著减少可训练参数。实验表明，该方法仅需约一半的微调参数，即可保持接近原始性能，下游任务表现几乎没有损失。
原文摘要 · Abstract (English)
In this paper, we introduce Symmetric Low-Rank Adapters, an optimized variant of LoRA with even fewer weights. This method utilizes Low-Rank Symmetric Weight Matrices to learn downstream tasks more efficiently. Traditional LoRA accumulates fine-tuning weights with the original pre-trained weights via a Singular Value Decomposition (SVD) like approach, i.e., model weights are fine-tuned via updates of the form $BA$ (where $B \in \mathbb{R}^{n\times r}$, $A \in \mathbb{R}^{r\times n}$, and $r$ is the rank of the merged weight matrix). In contrast, our approach, named SymLoRA, represents fine-tuning weights as a Spectral Decomposition, i.e., $Q \, diag(Λ)\, Q^T$, where $Q \in \mathbb{R}^{n\times r}$ and $Λ\in \mathbb{R}^r$. SymLoRA requires approximately half of the finetuning weights. Here, we show that this approach has negligible losses in downstream efficacy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。