用块循环矩阵降低大模型微调成本，存算开销大幅减少。
Block Circulant Adapter for Large Language Models
- 采用块循环矩阵与傅里叶变换结合，优化参数更新机制。
- 参数量比VeRA少14倍，比LoRA小16倍，计算量低32倍。
- 适合资源受限下高效微调大模型的场景，尤其关注存算效率者。
大语言模型微调因模型规模巨大而困难。基于傅里叶域的方法显示出降低微调成本的潜力。我们提出一种基于块循环矩阵的微调方法，结合稳定的训练启发式策略，利用循环矩阵与一维傅里叶变换的特性，显著降低存储和计算开销。实验表明，该方法在参数量上比VeRA少14倍、比LoRA小16倍，浮点运算量（FLOPs）比FourierFT低32倍，同时保持相近或更优的任务性能。该方法为频域微调大模型提供了有前景的新路径。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) is difficult due to their huge model size. Recent Fourier domain-based methods show potential for reducing fine-tuning costs. We propose a block circulant matrix-based fine-tuning method with a stable training heuristic to leverage the properties of circulant matrices and one-dimensional Fourier transforms to reduce storage and computation costs. Experiments show that our method uses $14\times$ less number of parameters than VeRA, $16\times$ smaller than LoRA and $32\times$ less FLOPs than FourierFT, while maintaining close or better task performance. Our approach presents a promising way in frequency domain to fine-tune large models on downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。