通过动态定位关键参数，实现高效高秩微调。
LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization
- 基于梯度稀疏性分析定位重要子网络进行优化
- 训练时间比LoRA减少27%，性能接近全量微调
- 适合资源受限但需高精度微调的场景
参数高效微调（PEFT）方法如LoRA通过引入低秩分解矩阵显著减少可训练参数。然而，现有方法在领域专精任务中执行大量矩阵乘法，导致计算效率低下且微调性能不佳。为此，我们提出LoSiA（低资源子网集成适配），一种创新方法，在训练过程中动态定位并优化关键参数。具体而言，它利用梯度稀疏性分析识别子网络，并将其作为可训练目标。该设计通过仅更新子网络参数实现有效高秩适配，减少额外矩阵乘法。我们还提出了LoSiA-Pro，其训练延迟比LoRA降低约27%。大量实验表明，该方法在领域专精和常识推理任务中，性能接近全量微调，同时训练时间最少。进一步分析显示，LoSiA还能减轻持续训练中的遗忘问题。代码已开源：https://github.com/KlozeWang/LoSiA。
原文摘要 · Abstract (English)
Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA, significantly reduce the number of trainable parameters by introducing low-rank decomposition matrices. However, existing methods perform extensive matrix multiplications in domain specialization tasks, resulting in computational inefficiency and sub-optimal fine-tuning performance. Hence, we propose LoSiA(Low-Resources Subnet Integration Adaptation), an innovative method that dynamically localizes and optimizes critical parameters during the training process. Specifically, it identifies a sub-network using gradient sparsity analysis and optimizes it as the trainable target. This design enables effective high-rank adaptation by updating only the sub-network parameters, reducing the additional matrix multiplication. We also present LoSiA-Pro, a faster implementation of LoSiA, which reduces the training latency by about $27\%$ compared to LoRA. Extensive evaluations show that our method achieves minimal performance drop compared to full fine-tuning, while requiring the least training time across domain specialization and common-sense reasoning tasks. Further analysis shows that LoSiA also reduces forgetting during continued training. The source code is available at https://github.com/KlozeWang/LoSiA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。