在低秩适配器中找到了能媲美全量参数的稀疏胜出网络。
The Quest for Winning Tickets in Low-Rank Adapters
- 发现低秩适配器内存在可独立训练的稀疏子网络。
- 稀疏度分布比具体权重选择更影响性能，最多减少87%参数量。
- 适合追求高效微调的模型部署与资源受限场景。
彩票策略假设（LTH）指出，过参数化神经网络中包含稀疏子网络（‘胜出票’），这些子网络在从零开始训练时能达到完整模型的性能。随着大型预训练模型微调的广泛应用，我们研究了LTH是否适用于参数高效微调（PEFT），特别是低秩适配（LoRA）方法。关键发现是：LTH在LoRA中成立，揭示了能匹配密集适配器性能的稀疏子网络。特别地，稀疏子网络的有效性更多取决于每层施加的稀疏程度，而非子网络中具体的权重。基于此洞察，我们提出Partial-LoRA，一种系统识别此类子网络并训练与任务相关子空间对齐的稀疏低秩适配器的方法。在8个视觉和12个语言任务（单任务与多任务设置）上的实验表明，Partial-LoRA将可训练参数量减少最多达87%，同时保持或提升准确率。结果不仅深化了对迁移学习及预训练与微调之间关系的理论理解，也为开发更高效的适配策略开辟了新路径。
原文摘要 · Abstract (English)
The Lottery Ticket Hypothesis (LTH) suggests that over-parameterized neural networks contain sparse subnetworks ("winning tickets") capable of matching full model performance when trained from scratch. With the growing reliance on fine-tuning large pretrained models, we investigate whether LTH extends to parameter-efficient fine-tuning (PEFT), specifically focusing on Low-Rank Adaptation (LoRA) methods. Our key finding is that LTH holds within LoRAs, revealing sparse subnetworks that can match the performance of dense adapters. In particular, we find that the effectiveness of sparse subnetworks depends more on how much sparsity is applied in each layer than on the exact weights included in the subnetwork. Building on this insight, we propose Partial-LoRA, a method that systematically identifies said subnetworks and trains sparse low-rank adapters aligned with task-relevant subspaces of the pre-trained model. Experiments across 8 vision and 12 language tasks in both single-task and multi-task settings show that Partial-LoRA reduces the number of trainable parameters by up to 87\%, while maintaining or improving accuracy. Our results not only deepen our theoretical understanding of transfer learning and the interplay between pretraining and fine-tuning but also open new avenues for developing more efficient adaptation strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。