只更新MLP层1%的关键参数，显著降低微调内存占用
SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers
- 将梯度转换为稀疏结构，仅保留1%重要参数进行更新
- 在相同内存下，优于LoRA和MeProp等主流方法
- 特别适合大规模模型中对MLP层的高效微调
Transformer模型性能的提升依赖于参数量和文本长度的增加，导致全模型微调成为高内存消耗任务。当前高效的参数高效微调（PEFT）方法多针对注意力模块，常忽略包含约一半参数的MLP模块。本文提出一种新方法SparseGrad，专用于高效微调MLP层。该方法将层梯度映射到仅约1%元素显著的稀疏空间，通过稀疏化梯度实现参数更新数量的大幅减少。我们在BERT和RoBERTa的自然语言理解任务，以及LLaMa-2的问答任务上验证了该方法。在相同内存开销下，SparseGrad显著优于LoRA和MeProp等主流SOTA PEFT方法。
原文摘要 · Abstract (English)
The performance of Transformer models has been enhanced by increasing the number of parameters and the length of the processed text. Consequently, fine-tuning the entire model becomes a memory-intensive process. High-performance methods for parameter-efficient fine-tuning (PEFT) typically work with Attention blocks and often overlook MLP blocks, which contain about half of the model parameters. We propose a new selective PEFT method, namely SparseGrad, that performs well on MLP blocks. We transfer layer gradients to a space where only about 1\% of the layer's elements remain significant. By converting gradients into a sparse structure, we reduce the number of updated parameters. We apply SparseGrad to fine-tune BERT and RoBERTa for the NLU task and LLaMa-2 for the Question-Answering task. In these experiments, with identical memory requirements, our method outperforms LoRA and MeProp, robust popular state-of-the-art PEFT approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。