arXiv:2509.17688cs.CLcs.CV2025-09EMNLP被引 3

TASO通过任务对齐稀疏优化,让LoRA更高效省参数。

TASO: Task-Aligned Sparse Optimization for Parameter-Efficient Model Adaptation

  • 基于预训练权重重要性识别任务核心区域,构建稀疏结构
  • 参数量与rank=1的LoRA相当,性能却全面超越
  • 适合追求高效微调的开发者,尤其关注参数压缩场景

LoRA因其简单有效已成为最流行的参数高效微调方法之一。然而,大量研究指出,LoRA常引入显著的参数冗余,不仅增加可训练参数数量,还削弱微调效果。由于难以准确识别LoRA中的冗余参数,如何高效精准地消除冗余仍是难题。本文提出TASO,一种利用预训练模型权重的重要性信息来缓解LoRA冗余的方法。具体而言,我们估计下游任务中的参数重要性,并根据重要性得分分布识别任务相关的核心区域。这些核心区域的位置信息被用于确定LoRA模块的稀疏结构,从而在微调前实现冗余消除。该方法显著减少了任务适配所需的可训练参数数量,同时为LoRA冗余消除提供了新的任务对齐视角。实验表明,在与rank $r = 1$ 的标准LoRA相当的参数预算下,TASO在多个任务上持续优于基准方法,实现了强劲的微调性能并有效去除冗余参数。

原文摘要 · Abstract (English)

LoRA has become one of the most widely used parameter-efficient fine-tuning methods due to its simplicity and effectiveness. However, numerous studies have shown that LoRA often introduces substantial parameter redundancy, which not only increases the number of trainable parameters but also hinders the effectiveness of fine-tuning. Since identifying redundant parameters in LoRA is inherently difficult, how to eliminate them efficiently and accurately remains a challenging problem. In this paper, we propose TASO, a redundancy reduction method that leverages importance information from the pretrained model's weights to mitigate LoRA redundancy. Specifically, we estimate parameter importance on downstream tasks and identify task-specific core regions based on the distribution of importance scores. The location information of these core regions is then used to determine the sparse structure of LoRA modules, enabling redundancy removal before fine-tuning. Our approach significantly reduces the number of trainable parameters required for task adaptation, while providing a novel task-aligned perspective for LoRA redundancy reduction. Experimental results demonstrate that, with a parameter budget comparable to LoRA with rank $r = 1$, TASO consistently outperforms standard LoRA across multiple tasks, achieving strong fine-tuning performance while effectively eliminating redundant parameters.

LoRA参数效率稀疏优化微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。