用循环与对角矩阵分解，大幅降低大模型微调的参数和计算量。
Parameter-Efficient Fine-Tuning with Circulant and Diagonal Vectors
- 通过交错循环矩阵与对角矩阵的乘积分解，压缩微调参数。
- 在多个任务上性能相近或更优，参数量与浮点运算减少超60%。
- 适合资源受限场景，如移动端或边缘设备部署大模型。
基础模型在多个领域取得巨大成功，但其庞大的计算与存储开销使其微调困难且实用性受限。近期研究发现，在傅里叶域中训练可有效提升微调性能并减少参数量。本文提出进一步降低复杂度的方法：通过交错循环矩阵与对角矩阵的乘积进行因子分解，并针对非方阵微调权重问题，将循环矩阵分块处理。该方法避免构建权重变化矩阵，仅使用一维快速傅里叶变换（1D FFT）替代二维FFT。实验表明，本方法在多种任务上达到相似或更优性能，同时显著降低浮点运算次数（FLOPs）和可训练参数数量。
原文摘要 · Abstract (English)
Foundation models have achieved tremendous success in different domains. However, their huge computation and storage complexity make these models difficult to fine-tune and also less applicable in practice. Recent study shows training in Fourier domain can be an effective fine-tuning method in terms of both model performance and number of training parameters. In this work, we propose to further reduce the complexity by the factorization through the product of interleaved circulant and diagonal matrices. In addition, we address the case of non-square fine-tuning weights by partitioning the circulant matrix into blocks. Our method avoids the construction of weight change matrix and utilizes 1D fast Fourier transform (FFT) instead of 2D FFT. Experimental results show that our method achieves similar or better performance across various tasks with much less floating-point operations (FLOPs) and the number of trainable parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。