arXiv:2604.19520cs.AI2026-04被引 1

通过相似性与差异性联合评估,实现大模型层剪枝的稳定高效。

SimDiff: Depth Pruning via Similarity and Difference

论文配图:SimDiff: Depth Pruning via Similarity and Difference
图 1 · 摘自论文原文
  • 从表征相似性和变换差异两个角度综合判断层重要性
  • 在0.5B到13B模型上均显著优于现有方法,7B模型剪枝25%仍保持91%性能
  • 适合追求模型轻量化且需稳定性能的部署场景

深度剪枝通过识别并移除冗余层来提升大语言模型的部署效率。目前普遍采用余弦距离衡量层间相似性作为判断标准,但仅依赖这一维启发式方法会导致性能波动甚至灾难性崩溃。为此,我们提出SimDiff,一种从表征相似性和变换差异两个正交视角联合评估层重要性的新准则。差异性通过两种不同度量方式量化:对异常值敏感的MSSD,用于识别作出关键修正的层;以及稳健测量平均贡献的MASD。在0.5B至13B参数量的多种模型上进行的大量实验表明,SimDiff在不同剪枝比例下均显著优于现有先进基线。特别地,在LLaMA2-7B上以25%剪枝率保留超过91%性能,并在LLaMA3.1-8B上剪枝12层实现最高1.49倍推理加速。此外,剪枝后模型可通过极少微调有效恢复。

原文摘要 · Abstract (English)

Depth pruning improves the deployment efficiency of large language models (LLMs) by identifying and removing redundant layers. A widely accepted standard for this identification process is to measure the similarity between layers using cosine distance. However, we find that methods relying solely on this one-dimensional heuristic can exhibit unpredictable performance and even catastrophic collapse across different architectures. To address this issue, we propose SimDiff, a novel layer importance criterion that jointly evaluates layers from two orthogonal perspectives: representational similarity and transformation difference. The difference is quantified using two distinct metrics: MSSD, which is sensitive to outliers and identifies layers that make decisive corrections, and MASD, which robustly measures a layer's average contribution. Extensive experiments on multiple models ranging from 0.5B to 13B parameters demonstrate that SimDiff significantly outperforms state-of-the-art baselines across various pruning ratios. Notably, our method retains over 91% of LLaMA2-7B's performance at a 25% pruning ratio and achieves up to a 1.49x inference speedup when pruning 12 layers on LLaMA3.1-8B. We also show that pruned models can be effectively recovered with minimal fine-tuning.

模型剪枝大模型优化层重要性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。