发现Transformer模型各层差异巨大，可按重要性分配算力，省54%成本。
Anatomical Heterogeneity in Transformer Language Models
- 分析30层小模型，发现层间权重有强数学规律但不可直接替换。
- 核心层（8-11层）删除后损失超6万%，反向层删了反而更好。
- 按层重要性分配训练资源，同等参数下速度更快、损失更低。
当前Transformer语言模型对所有层使用相同的计算预算，隐含假设层是同质的。我们通过分析一个30层、135M参数的因果语言模型SmolLM2-135M，采用五种诊断指标：权重可预测性（R²）、消融退化、恢复速度、权重扰动鲁棒性及结构分析，揭示了显著的解剖异质性：（1）层权重呈现强数学规律（R²=0.91），具有普遍振荡差值模式（相关性≈-0.50），但预测权重因非线性误差累积导致灾难性失败；（2）层重要性跨度达10⁷倍，核心层（L8-L11）删除后困惑度提升高达+63,419%，而反向层（L14、L17）移除反而提升性能；（3）恢复速度与层重要性相关，表明训练需求不同；（4）五种权重操作策略中仅权重缩放（alpha=0.9）能保持模型质量；（5）基于层重要性的增长型训练（Growth Transformer Training）实现约54%成本降低。概念验证实验表明，在相同参数量下，该方法验证损失比均匀训练低4.7倍，且快13%。
原文摘要 · Abstract (English)
Current transformer language models are trained with uniform computational budgets across all layers, implicitly assuming layer homogeneity. We challenge this assumption through empirical analysis of SmolLM2-135M, a 30-layer, 135M-parameter causal language model, using five diagnostic metrics: weight predictability (R2), ablation degradation, recovery speed, weight manipulation robustness, and structural analysis. We find profound anatomical heterogeneity: (1) Layer weights follow strong mathematical regularity (R2 = 0.91) with a universal oscillatory delta pattern (correlation ~= -0.50), yet predicted weights cause catastrophic failure due to nonlinear error accumulation. (2) Layer importance spans a 10^7 range, from a critical core (L8-11, up to +63,419% PPL degradation) to anti-layers (L14, L17) whose removal improves performance. (3) Recovery speed correlates with layer importance, indicating differential training requirements. (4) Only weight scaling (alpha = 0.9) preserves model quality among five tested manipulation strategies. (5) Growth Transformer Training, allocating budget by layer importance, achieves ~54% cost reduction. A proof-of-concept experiment confirms this: 4.7x lower validation loss than uniform training at identical parameter count, while being 13% faster.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。