提出新方法在不依赖SVD的情况下实现高效低秩微调，性能优于传统LoRA。
Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC
- 将LoRA优化转化为近端子空间问题，用交替最小二乘法高效求解
- 在多个数据集上达到或超过LoRA基线，计算开销小且无需存储完整SVD矩阵
- 适用于需高效率微调的大模型，尤其适合资源受限场景
低秩适配（LoRA）通过在冻结权重上学习低秩更新来微调大模型，显著减少可训练参数和内存占用。本文填补了全步梯度下降与低秩投影（如SVDLoRA）之间的差距。提出一种名为LoRSum的内存高效的子程序，将LoRA优化建模为近端子问题，并通过交替最小二乘法求解，证明其等价于隐式块幂法。该方法涵盖若干近期提出的预条件化方法作为特例，且可扩展用于低秩动量更新。为进一步支持预条件梯度下降，提出一种缩放变体，使用如K-FAC、Shampoo等结构化度量，并证明仅存储其对角线即可保持良好性能，同时保持内存效率。在合成任务、CIFAR-100以及GLUE、SQuAD v2、WikiText-103语言模型微调任务上的实验表明，本方法在适度计算开销下可匹配或超越标准LoRA，避免完整矩阵的SVD投影，维持LoRA级参数效率。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights, dramatically reducing trainable parameters and memory. In this work, we address the gap between training with full steps with low-rank projections (SVDLoRA) and LoRA fine-tuning. We propose LoRSum, a memory-efficient subroutine that closes this gap for gradient descent by casting LoRA optimization as a proximal sub-problem and solving it efficiently with alternating least squares updates, which we prove to be an implicit block power method. We recover several recently proposed preconditioning methods for LoRA as special cases, and show that LoRSum can also be used for updating a low-rank momentum. In order to address full steps with preconditioned gradient descent, we propose a scaled variant of LoRSum that uses structured metrics such as K-FAC and Shampoo, and we show that storing the diagonal of these metrics still allows them to perform well while remaining memory-efficient. Experiments on a synthetic task, CIFAR-100, and language-model fine-tuning on GLUE, SQuAD v2, and WikiText-103, show that our method can match or improve LoRA baselines given modest compute overhead, while avoiding full-matrix SVD projections and retaining LoRA-style parameter efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。