修复剪枝大模型的层间激活不匹配，无需重新训练即可恢复性能。
Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMs
- 通过小规模校准集推导线性映射，直接对齐被剪枝层的激活分布。
- 在多个大模型和剪枝策略下，准确率与困惑度均优于现有无训练方法。
- 适用于追求高效推理的场景，特别适合已剪枝模型的性能修复。
层剪枝会移除大型语言模型中的整个Transformer解码器块，导致下一存活层接收到的隐藏状态分布与其训练时预期不符，引发显著性能下降。本文提出Ghosted Layers，一种无需训练的恢复模块，通过求解边界激活对齐问题来解决该问题。方法基于少量校准数据，推导出闭式最优线性算子，重建因剪枝引入的激活差异。该解对应于对齐目标的无约束最优解，而现有方法受限于有限算子子空间的约束解。在多个LLM主干和剪枝策略上的实验表明，本方法持续优于先前的无训练基线，在保持层剪枝效率优势的同时提升准确率与困惑度。官方代码仓库：https://github.com/daniel-eai/ghosted_layers_official_repository/
原文摘要 · Abstract (English)
Layer pruning removes entire Transformer decoder blocks from large language models, but introduces a mismatch between the hidden state received by the next surviving layer and the distribution it was trained to process, leading to significant performance degradation. We propose Ghosted Layers, a training-free recovery module that addresses this issue by solving a boundary activation alignment problem. Our method derives a closed-form optimal linear operator from a small calibration set to reconstruct the activation discrepancy introduced by the pruned layers. We show that this solution corresponds to the unconstrained optimum of the alignment objective, whereas existing methods are restricted to constrained solutions over limited operator subspaces. Experiments across multiple LLM backbones and pruning strategies demonstrate that our method consistently improves accuracy and perplexity over prior training-free baselines, while preserving the efficiency gains of layer pruning. Official code repository: https://github.com/daniel-eai/ghosted_layers_official_repository/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。