渐进式增深模型能突破深度诅咒，提升推理能力
Do Depth-Grown Models Overcome the Curse of Depth? An In-Depth Analysis
- 通过渐进堆叠中间层优化深度利用效率
- 实验证明增深后残差流结构更合理，计算模块可置换
- 适合关注大模型高效训练与推理的从业者
在训练过程中逐步增加Transformer深度，不仅能降低训练成本，还能提升推理性能，如MIDAS（Saunshi等，2024）所示。然而，目前尚缺乏对这一提升机制的深入理解。本文揭示了这种增益与近期研究发现的关联：在非增深、预层归一化的Transformer中，后半部分层对最终输出分布的贡献远小于前半部分——即深度诅咒（Sun等，2025；Csordás等，2025）。通过逐层分析，我们证明渐进式中间堆叠能更有效地利用模型深度，改变残差流结构，并促进可置换计算块的形成。此外，我们提出一种MIDAS的轻量级改进，进一步提升了下游推理基准表现。整体而言，该工作说明渐进式深度增长可促成独特的计算电路，克服传统非增深模型中深度利用率低的问题。
原文摘要 · Abstract (English)
Gradually growing the depth of Transformers during training can not only reduce training cost but also lead to improved reasoning performance, as shown by MIDAS (Saunshi et al., 2024). Thus far, however, a mechanistic understanding of these gains has been missing. In this work, we establish a connection to recent work showing that layers in the second half of non-grown, pre-layernorm Transformers contribute much less to the final output distribution than those in the first half - also known as the Curse of Depth (Sun et al., 2025, Csordás et al., 2025). Using depth-wise analyses, we demonstrate that growth via gradual middle stacking yields more effective utilization of model depth, alters the residual stream structure, and facilitates the formation of permutable computational blocks. In addition, we propose a lightweight modification of MIDAS that yields further improvements in downstream reasoning benchmarks. Overall, this work highlights how the gradual growth of model depth can lead to the formation of distinct computational circuits and overcome the limited depth utilization seen in standard non-grown models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。