用梯度指导剪枝，让大模型微调又快又省。
GradPruner: Gradient-Guided Layer Pruning Enabling Efficient Fine-Tuning and Inference for LLMs
- 根据微调初期的累积梯度评估层重要性，动态剪枝。
- 剪掉40%参数，精度仅下降0.99%。
- 适合追求高效微调与推理的工程应用者。
大语言模型在下游数据上微调通常耗时且昂贵。结构化剪枝虽能提升推理效率，但常需额外训练、知识蒸馏或结构搜索，难以实现高效微调。为此，我们提出GradPruner，利用微调初期各参数的累积梯度计算初始梯度信息累积矩阵（IGIA-Matrix），评估层重要性并执行剪枝。剪枝后层按相同符号元素合并，减少符号差异干扰。我们在两个LLM和八个下游数据集（涵盖医疗、金融及通用基准任务）上进行了广泛实验。结果表明，GradPruner实现了40%的参数减少，精度仅下降0.99%。代码已公开。
原文摘要 · Abstract (English)
Fine-tuning Large Language Models (LLMs) with downstream data is often considered time-consuming and expensive. Structured pruning methods are primarily employed to improve the inference efficiency of pre-trained models. Meanwhile, they often require additional time and memory for training, knowledge distillation, structure search, and other strategies, making efficient model fine-tuning challenging to achieve. To simultaneously enhance the training and inference efficiency of downstream task fine-tuning, we introduce GradPruner, which can prune layers of LLMs guided by gradients in the early stages of fine-tuning. GradPruner uses the cumulative gradients of each parameter during the initial phase of fine-tuning to compute the Initial Gradient Information Accumulation Matrix (IGIA-Matrix) to assess the importance of layers and perform pruning. We sparsify the pruned layers based on the IGIA-Matrix and merge them with the remaining layers. Only elements with the same sign are merged to reduce interference from sign variations. We conducted extensive experiments on two LLMs across eight downstream datasets. Including medical, financial, and general benchmark tasks. The results demonstrate that GradPruner has achieved a parameter reduction of 40% with only a 0.99% decrease in accuracy. Our code is publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。