深度剪枝后用轻量适配器修复分布偏移,无需训练即可恢复模型性能。
SHIFT-LLM: Distribution Shift Correction in Depth-Pruned LLMs

- 在剪枝位置插入线性残差适配器,保留原始路径并添加轻量校正。
- 仅需少量样本通过最小二乘法校准,最高提升15.7分准确率。
- 无需反向传播,支持低秩压缩和高效微调,适合部署优化场景。
深度剪枝通过移除整个Transformer层降低大语言模型的推理开销,但破坏了下游层期望的隐藏状态分布,导致显著精度下降。本文提出SHIFT-LLM,一种无需训练的剪枝后修正框架,在每个剪枝位置插入线性残差适配器(LRA)。每个LRA保留原残差块的身份路径,并添加轻量仿射残差校正。该校正通过小规模保留集上的闭式最小二乘回归进行校准,无需梯度计算,以逼近被剪枝层缺失的残差更新。结合保留的身份路径,LRA输出近似于原层产生的隐藏状态,从而缓解因层移除引入的分布失配,同时避免被移除层的昂贵注意力与前馈计算。所生成的LRA支持低秩分解及连续剪枝层的精确合并,实现额外压缩;并与参数高效微调自然兼容,可进一步恢复剪枝模型单独微调无法达到的性能。在五个模型家族、六种层选择策略和七个零样本基准上的实验表明,SHIFT-LLM在多数配置下均能一致恢复深度剪枝造成的精度损失,最大在Llama-3.1-8B-Instruct上提升15.7点,仅需数百个校准样本且无梯度计算。
原文摘要 · Abstract (English)
Depth pruning removes entire Transformer blocks to reduce the inference cost of large language models, but disrupts the hidden-state distributions expected by downstream layers, leading to significant accuracy loss. We introduce SHIFT-LLM, a training-free post-pruning correction framework that inserts a Linear Residual Adapter (LRA) at each pruning site. Each LRA preserves the identity pathway of the original residual block and adds a lightweight affine residual correction. This correction is calibrated via closed-form least-squares regression on a small held-out set, without gradient computation, to approximate the missing residual update produced by the pruned block. Together with the preserved identity pathway, the resulting LRA output approximates the hidden state produced by the original block, thereby mitigating the distributional mismatch introduced by layer removal while avoiding the expensive attention and feed-forward computations of the removed blocks. The resulting LRAs support low-rank factorization and exact merging across consecutive pruned layers for additional compression, and combine naturally with parameter-efficient fine-tuning for further recovery beyond fine-tuning the pruned model alone. Experiments on five model families, six layer-selection criteria, and seven zero-shot benchmarks show that SHIFT-LLM consistently recovers accuracy lost to depth pruning across most configurations, achieving gains up to +15.7 points on Llama-3.1-8B-Instruct while requiring only a few hundred calibration samples and no gradient computation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。