arXiv:2603.20991cs.LGcs.AI2026-03

提出误差传播度量rho,指导模型压缩中层删减与权重剪枝。

Structural Sensitivity in Compressed Transformers: Relative Error Propagation and Layer Removal

  • 定义rho值衡量每层输入输出误差比,判断误差是衰减还是放大。
  • 早层压缩影响更大,深度减少剪枝比均匀剪枝效果好20%以上。
  • 无需训练即可快速筛选可删层数,适合部署优化场景。

压缩Transformer权重可降低大语言模型部署成本,但每层压缩引入误差,且误差随信号传递累积,机制尚不明确。本文直接测量各层输出与输入误差的比率(称rho):rho < 1表示层吸收误差,>1则放大。在六种Transformer(参数量117M至8B)上计算rho,得出三结论:(i) 层t的误差向下传播倍数为后续rho值乘积,预测表征漂移(斯皮尔曼相关r = -0.44,p < 10^-4),解释为何压缩早期层损害更大,且深度递减稀疏策略优于均匀策略;尽管如此,模型宽度与冗余度对性能影响大于rho本身。(ii) 同层内,朴素剪枝显示组件敏感度差异达约600倍,而激活感知剪枝(Wanda)将差异缩小至3-7倍,且不同架构间重要性排序反转,表明固定重要性评分无法迁移。(iii) 对于深度剪枝,根据rho偏离1的程度排序只需两轮前向传播,比ShortGPT的块影响方法降低1.6倍困惑度(八层删除时),物理删除带来1.22倍运行速度提升。结合两者指标表现最佳(困惑度14.2,LLaMA-2-7B下游准确率60.0%)。十二个Lean 4范数不等式提供机器可验证的每矩阵误差边界。收缩轮廓由此成为无需训练的压缩决策工具:用于层内压缩位置选择与层删减顺序。

原文摘要 · Abstract (English)

Compressing transformer weights makes large language models cheaper to deploy. But each layer's compression introduces an error. These errors accumulate as the signal passes through later layers, and how they accumulate is not well understood. We measure this directly: at each layer, we take the ratio of output to input error, calling it rho. A value below one means the layer absorbs the error; above one means it grows. Computing rho on six transformers (117M to 8B parameters) yields three findings. (i) Errors at layer t scale downstream by the product of later rho values, predicting representation drift (Spearman r = -0.44, p < 10^-4). This explains why compressing early layers hurts more than late ones, and why depth-decreasing sparsity schedules outperform uniform ones. Across architecture families, however, model width and redundancy matter more than rho alone. (ii) Within a layer, naive pruning shows a ~600x spread in component sensitivity. Activation-aware pruning (Wanda) shrinks this to 3-7x; the ranking reverses across architectures, so fixed importance scores do not transfer. (iii) For depth pruning, ranking layers by how far rho is from one takes two forward passes. It beats ShortGPT's Block Influence with 1.6x lower perplexity at eight layers removed, and physical deletion delivers 1.22x wall-clock speed-up. A blend of the two criteria does best (perplexity 14.2, 60.0% downstream accuracy on LLaMA-2-7B). Twelve Lean 4 norm inequalities provide machine-checked per-matrix error bounds. The contraction profile thus gives a training-free instrument for two decisions: where to compress within layers, and which to remove.

模型压缩误差传播剪枝策略Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。