分析大模型压缩的结构性瓶颈，发现方差不等于重要性，线性随深度增强。
Variance Is Not Importance: Structural Analysis of Transformer Compressibility Across Model Scales
- 通过多类实验验证压缩关键结构特性，识别五项核心规律。
- 高方差方向与预测无关，投影保留90%方差但仅微增困惑度1.71。
- 适合研究模型压缩机制或设计自适应推理的开发者参考。
我们对 GPT-2（1.24亿参数）和 Mistral 7B（72.4亿参数）进行了超过40次系统性实验，研究了谱压缩、块级函数替换、基于旋转的量化、激活几何结构及自适应提前退出等压缩方法。识别出五个与压缩相关的结构性特征：(1) 方差非重要性：高方差激活方向与预测方向相关性极低（CCA测得约96%不相关），投影到这些子空间可保留超90%方差，但困惑度仅轻微上升；(2) 块线性为条件性：在正确上游分布下，变换器块近似线性（GPT-2 R²~0.95，Mistral第31块R²~0.93），修改早期块会引发分布偏移，破坏下游近似；(3) 重建墙：将权重分解为量化组件的方法因交叉项放大误差，直接量化更优；(4) 线性随深度增强：Mistral 7B从第0块（R²=0.17）到第31块（R²=0.93），体现非线性构建与线性精调的分工；(5) 约30%的输入令牌计算成本较低，由退出头与KL敏感度验证。单块线性替换可在Mistral 7B末尾实现34倍压缩，困惑度仅升1.71，而多块替换因残差误差累积与分布偏移失败。结果表明静态后训练压缩存在根本局限，建议采用逐令牌自适应计算。
原文摘要 · Abstract (English)
We present a systematic empirical study of transformer compression through over 40 experiments on GPT-2 (124M parameters) and Mistral 7B (7.24B parameters). Our analysis covers spectral compression, block-level function replacement, rotation-based quantization, activation geometry, and adaptive early exit. We identify five structural properties relevant to compression. (1) Variance is not importance: high-variance activation directions are approximately 96 percent uncorrelated with predictive directions (measured via CCA), and projecting onto these subspaces preserves over 90 percent of variance while degrading perplexity. (2) Block linearity is conditional: transformer blocks are approximately linear (R^2 ~ 0.95 on GPT-2, 0.93 on Mistral block 31) only under the correct upstream distribution; modifying earlier blocks induces distribution shift that degrades downstream approximations. (3) The reconstruction wall: approaches that factor weights into quantized components amplify errors through cross-terms, making direct quantization strictly superior. (4) Linearity increases with depth: Mistral 7B exhibits a progression from R^2 = 0.17 (block 0) to R^2 = 0.93 (block 31), indicating a division between nonlinear feature construction and linear refinement. (5) Approximately 30 percent of tokens are computationally easy, confirmed via exit heads and KL divergence sensitivity. We demonstrate that single-block linear replacement achieves 34x compression with a 1.71 perplexity increase on the final block of Mistral 7B, while multi-block replacement fails due to residual error accumulation and distribution shift. These findings suggest fundamental limits to static post-training compression and motivate adaptive, per-token computation as a more effective direction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。