深度剪枝效果关键在校准配置，而非复杂搜索算法。
Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning
- 从功能角度重新审视层冗余，强调校准配置的作用。
- 简单的一次性方法与复杂搜索结果接近，性能差异小。
- 适合关注模型压缩效率的研究者和工程落地团队。
深度剪枝通过移除Transformer块提升大语言模型的推理效率。以往工作通常将层冗余视为预训练网络的固有结构特性，侧重于重要性度量和搜索算法以识别可移除层。本研究从功能视角实证考察深度剪枝，评估多个代表性LLM家族在不同校准配置和多种搜索算法下的表现，发现不同配置产生不同剪枝模式。在固定校准配置下，复杂搜索算法对性能的提升微乎其微，与简单的一次性方法收敛至相似的剪枝子集。总体而言，校准配置对剪枝模式和校准困惑度的影响远大于搜索算法选择，且在下游推理准确率方差中贡献相当。这表明未来剪枝工作应优先优化校准配置，而非追求搜索复杂度。
原文摘要 · Abstract (English)
Depth pruning improves the inference efficiency of large language models by removing Transformer blocks. Prior work typically treats layer redundancy as an inherent structural property of pretrained networks, emphasizing importance criteria and search algorithms to identify removable layers. In this study, we empirically investigate depth pruning from a functional perspective. Evaluating representative LLM families across diverse calibration configurations and multiple search algorithms, we show that different configurations produce different pruning patterns. Furthermore, under a fixed calibration configuration, complex search algorithms yield marginal performance improvements over simple one-shot methods, converging to similar pruned subsets. Overall, our results suggest that the calibration configuration plays a substantially larger role than the choice of search algorithm in shaping pruning patterns and calibration perplexity, while contributing comparably to variance in downstream reasoning accuracy. This indicates that future pruning efforts may benefit from prioritizing the calibration configuration over search complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。