只更新Transformer权重矩阵的行和列，实现高效微调大模型。
RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates
- 仅更新权重矩阵的少量行和列，降低参数量
- 在多个大模型上达到媲美顶尖方法的准确率
- 适合资源有限但需高效微调大模型的研究者
我们提出RoCoFT，一种基于仅更新Transformer权重矩阵中少量行和列的参数高效微调方法。在BERT、RoBERTa等中型模型，以及Bloom-7B、Llama2-7B和Llama2-13B等大型语言模型上进行大量实验，结果表明该方法在保持或优于当前最优参数高效微调（PEFT）方法的同时，具备更低的内存与计算开销。通过神经正切核理论工具，我们实证分析了方法有效性：使用受限行/列参数构建的核，数值上接近全参数核，并实现相当的分类性能。消融实验进一步探究了行/列选择策略及最优秩对方法效果的影响。
原文摘要 · Abstract (English)
We propose RoCoFT, a parameter-efficient fine-tuning method for large-scale language models (LMs) based on updating only a few rows and columns of the weight matrices in transformers. Through extensive experiments with medium-size LMs like BERT and RoBERTa, and larger LMs like Bloom-7B, Llama2-7B, and Llama2-13B, we show that our method gives comparable or better accuracies than state-of-art PEFT methods while also being more memory and computation-efficient. We also study the reason behind the effectiveness of our method with tools from neural tangent kernel theory. We empirically demonstrate that our kernel, constructed using a restricted set of row and column parameters, are numerically close to the full-parameter kernel and gives comparable classification performance. Ablation studies are conducted to investigate the impact of different algorithmic choices, including the selection strategy for rows and columns as well as the optimal rank for effective implementation of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。