保护关键层提升压缩效率，让小模型更省参数还更好用。
AutoCompress: Critical Layer Isolation for Efficient Transformer Compression
- 识别出第一层是任务关键层，重要性超其他层60倍。
- 仅用1438万参数实现接近原模型的204.5困惑度。
- 专为关键层保留完整维度，适合追求高效压缩的研究者。
我们提出AutoCompress，一种基于实证发现的Transformer压缩方法：在小型Transformer中，第0层承载了远超其他层的任务关键信息，其基于NTK的重要性得分高达3.6，而其余层最高仅为0.054，差距超过60倍。据此，我们设计了关键层隔离（CLI）架构：保持第0层全维，中间层通过可学习瓶颈压缩，最后一层恢复全维。应用于GPT-2 Medium（3.548亿参数），CLI-GPT2在WikiText-103上达到204.5困惑度，仅需1.438亿参数——压缩比达2.47倍，参数减少59.5%。消融实验表明，同等规模的均匀瓶颈基线在相同训练条件下仅得571.8困惑度，证明保护第0层的架构选择才是性能提升主因。代码与检查点已公开。
原文摘要 · Abstract (English)
We present AutoCompress, a transformer compression method motivated by an empirical finding: in small transformers, Layer 0 carries disproportionately high task-critical information, with an NTK-based importance score of 3.6 compared to a maximum of 0.054 for all other layers -- a gap of over 60x. Based on this finding, we propose Critical Layer Isolation (CLI), an architecture that protects Layer 0 at full dimensionality, compresses all intermediate layers through a learned bottleneck, and restores the full dimension at the final layer. Applied to GPT-2 Medium (354.8M parameters), CLI-GPT2 achieves 204.5 perplexity on WikiText-103 with only 143.8M parameters -- a 2.47x compression ratio and 59.5% parameter reduction. Crucially, an ablation study demonstrates that a uniform bottleneck baseline of comparable size achieves only 571.8 perplexity under identical training conditions, confirming that the architectural decision to protect Layer 0 -- rather than simply reducing model size -- is the primary driver of performance. Code and checkpoints are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。