根据层间表征相似性，智能剪裁大模型深度以提升推理效率。
Locality-Aware Redundancy Pruning for LLM Depth Compression
- 基于层间表征相似性动态识别冗余层，避免固定剪裁策略。
- 在多个模型上实现更低困惑度与更高任务准确率。
- 无需训练，适合快速部署的模型压缩场景。
大型语言模型在深度方向上存在表征冗余，深度剪裁是提升推理效率的有效方法。现有一次性剪裁方法依赖局部层重要性或对架构的固定冗余假设。本文提出无训练的一次性深度剪裁框架 LoRP（Locality-Aware Redundancy Pruning），其核心为表征局部性。我们发现层间冗余可呈局部集中或全局分布，取决于模型架构。为此引入表征局部性评分（RLS），基于全局层间隐藏状态相似性计算。通过小规模校准集，LoRP 计算层对相似性，按表征相似性聚类，并依据簇内残余冗余分配剪裁量。跨多种大模型家族的实验表明，该方法在困惑度和下游任务准确率上均有提升。
原文摘要 · Abstract (English)
Large language models are known to contain representational redundancy across network depth, making depth pruning an effective approach for improving inference efficiency. Existing one-shot pruning methods rely on local layer importance or fixed redundancy assumptions across architectures. We propose Locality-Aware Redundancy Pruning (LoRP), a training-free one-shot depth pruning framework guided by representation locality. We show that inter-layer redundancy can be either localized or globally distributed depending on the LLM architecture. To characterize this phenomenon, we introduce Representation Locality Score (RLS), derived from global inter-layer hidden-state similarity. Using a small calibration set, LoRP computes pairwise layer similarity, clusters layers by representational similarity, and allocates pruning according to residual intra-cluster redundancy. Experiments across diverse LLM families show improvements in both perplexity and downstream task accuracy. Official github repository: https://github.com/daniel-eai/LoRP-Locality-Aware-Redundancy-Pruning/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。