只随机选少量参数训练，效果竟不输主流微调方法。
Sparsity May Be All You Need: Sparse Random Parameter Adaptation
- 随机挑选部分参数训练，无需低秩等结构假设。
- 使用相同可训练参数量时，性能媲美LoRA。
- 证明参数稀疏性比结构设计更关键，适合资源受限场景。
大模型对齐与任务适配的全量微调因模型规模增大而成本过高。参数高效微调（PEFT）通过仅训练少量参数来降低计算与内存开销。当前最流行的PEFT方法是低秩适应（LoRA），它冻结模型参数，在低秩矩阵中引入少量可训练参数。本文提出一种简单方法：仅随机选择一小部分模型参数进行训练，其余参数固定，无需额外假设如低秩结构。我们比较了该方法与其他PEFT方法及全参数微调的效率与性能。结果表明，在可训练参数数量相近时，该方法表现与LoRA相当。研究提示，决定PEFT性能的关键并非特定适配器结构，而是可训练参数的数量本身。
原文摘要 · Abstract (English)
Full fine-tuning of large language models for alignment and task adaptation has become prohibitively expensive as models have grown in size. Parameter-Efficient Fine-Tuning (PEFT) methods aim at significantly reducing the computational and memory resources needed for fine-tuning these models by only training on a small number of parameters instead of all model parameters. Currently, the most popular PEFT method is the Low-Rank Adaptation (LoRA), which freezes the parameters of the model and introduces a small set of trainable parameters in the form of low-rank matrices. We propose simply reducing the number of trainable parameters by randomly selecting a small proportion of the model parameters to train on, while fixing all other parameters, without any additional prior assumptions such as low-rank structures. In this paper, we compare the efficiency and performance of our proposed approach to other PEFT methods as well as full parameter fine-tuning. We find our method to be competitive with LoRA when using a similar number of trainable parameters. Our findings suggest that what truly matters for a PEFT technique to perform well is not necessarily the specific adapter structure, but rather the number of trainable parameters being used.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。