让高效微调更快更省显存，支持量化模型训练
Orthogonal Finetuning Made Scalable
- 改用输入中心计算方式，将复杂度从立方降至平方
- 训练速度提升10倍，显存降低3倍，性能不下降
- 可直接微调量化模型，比QLoRA更稳定高效
正交微调（OFT）在参数高效适应和防止灾难性遗忘方面表现优异，但其高运行时间和内存开销限制了实际应用。我们发现OFT的核心瓶颈在于基于权重的实现方式，依赖于具有三次复杂度的矩阵-矩阵乘法。为此，我们提出OFTv2，一种基于输入的重构方法，改用矩阵-向量乘法（即无矩阵计算），将计算复杂度降至二次。我们进一步引入Cayley-Neumann参数化，通过截断Neumann级数近似Cayley变换中的矩阵求逆，实现高效正交参数化。这些改进使OFTv2在不损失性能的前提下，实现最高10倍的训练速度提升和3倍的GPU内存降低。此外，我们将OFTv2扩展至支持量化基础模型的微调，并证明其在训练稳定性、效率和内存使用上均优于流行的QLoRA。
原文摘要 · Abstract (English)
Orthogonal finetuning (OFT) offers highly parameter-efficient adaptation while preventing catastrophic forgetting, but its high runtime and memory demands limit practical deployment. We identify the core computational bottleneck in OFT as its weight-centric implementation, which relies on costly matrix-matrix multiplications with cubic complexity. To overcome this, we propose OFTv2, an input-centric reformulation that instead uses matrix-vector multiplications (i.e., matrix-free computation), reducing the computational cost to quadratic. We further introduce the Cayley-Neumann parameterization, an efficient orthogonal parameterization that approximates the matrix inversion in the Cayley transform via a truncated Neumann series. These modifications allow OFTv2 to achieve up to 10x faster training and 3x lower GPU memory usage without compromising performance. In addition, we extend OFTv2 to support finetuning quantized foundation models and show that it outperforms the popular QLoRA in training stability, efficiency, and memory usage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。