用稀疏系数矩阵压缩参数更新，显著降低大模型微调内存占用。
Fine-Tuning of Transformer models with Frames
- 在融合帧基下用稀疏系数表示参数变化，减少存储开销。
- 相比现有方法，仅需少量可训练参数即可达到相当或更优性能。
- 适用于语言与视觉任务，特别适合资源受限场景的模型微调。
参数高效微调（PEFT）方法如低秩适应（LoRA）虽有效，但其内存开销随模型规模呈线性增长，为 $\mathcal{O}(dr)$，其中 $d$ 为隐藏维度，$r$ 为秩。本文提出 FrameFT，将参数更新 $ΔW$ 在融合帧基下建模为稀疏系数矩阵。融合帧可通过算法生成并跨层共享，仅需存储和优化稀疏系数，大幅降低内存占用。稀疏系数矩阵与融合帧本身的稀疏性带来显著计算优势，且理论分析给出了收敛性保证。我们在多种监督微调基准上评估，涵盖语言任务，并报告了在视觉模型上的应用。实验表明，FrameFT 性能可媲美或超越当前最优的 PEFT 方法，但所需可训练参数极少。
原文摘要 · Abstract (English)
Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scale with the size of the model, $\mathcal{O}(dr)$, where $d$ is the model's hidden dimension and $r$ is the rank. Our proposal, FrameFT, models the parameter update $ΔW$ with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling very efficient updates. Only the sparse coefficients of the basis expansion are stored/optimized, reducing the memory footprint. The sparse structure of the coefficient matrix in FrameFT and the sparsity in the Fusion Frames give large compute benefits, and our analysis provides formal convergence results. We evaluate the idea across a suite of supervised fine-tuning benchmarks, focusing on language tasks, but also report application to vision models. Our experiments show that FrameFT achieves performance on par with/exceeding state-of-the-art PEFT techniques, but needs far fewer trainable parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。