用预训练权重选骨架，低成本实现高秩微调
PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning
- 从预训练权重中挑选关键矩阵骨架，只学习小矩阵
- 在DROP和数学推理任务上显著优于LoRA，参数量更少
- 适合追求高效高精度微调的研究者和工程师
低秩适配（LoRA）及其变体因避免过高推理开销而受到关注，但仍面临两大挑战：（1）低秩假设的局限性；（2）初始化方法可能次优。为此，我们提出PMSS（Pre-trained Matrices Skeleton Selection），在保持低计算成本的同时实现高秩更新，并利用预训练权重中蕴含的语义与语言信息。其通过从预训练权重矩阵中选择骨架，仅需学习一个小型矩阵即可。实验表明，PMSS在多个任务上超越LoRA及其他微调方法，且可训练参数极少。尤其在复杂任务上表现突出：在DROP基准上，对LLaMA2-7B/13B分别提升3.4%/5.9%；在GSM8K数学推理任务上，对LLaMA2-7B、Mistral-7B和Gemma-7B分别提升12.89%、5.61%和3.11%。代码与模型即将开源。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) and its variants have recently gained much interest due to their ability to avoid excessive inference costs. However, LoRA still encounters the following challenges: (1) Limitation of low-rank assumption; and (2) Its initialization method may be suboptimal. To this end, we propose PMSS(Pre-trained Matrices Skeleton Selection), which enables high-rank updates with low costs while leveraging semantic and linguistic information inherent in pre-trained weight. It achieves this by selecting skeletons from the pre-trained weight matrix and only learning a small matrix instead. Experiments demonstrate that PMSS outperforms LoRA and other fine-tuning methods across tasks with much less trainable parameters. We demonstrate its effectiveness, especially in handling complex tasks such as DROP benchmark(+3.4%/+5.9% on LLaMA2-7B/13B) and math reasoning(+12.89%/+5.61%/+3.11% on LLaMA2-7B, Mistral-7B and Gemma-7B of GSM8K). The code and model will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。