arXiv:2603.23998cs.CL2026-03ACL被引 4

让Transformer训练时逐步增加深度,减少冗余计算。

Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping

  • 训练中动态调整深度,从深层向浅层渐进扩展注意力循环。
  • 相同性能下,额外计算量从16%-20%降至1%-3%。
  • 适合追求高效训练的模型优化研究者。

现有提升Transformer有效深度的方法主要依赖参数复用,通过递归执行扩展计算量。在此范式下,网络结构在训练过程中保持静态,额外计算深度均匀分配给整个模块,导致训练期间存在大量计算冗余。相比之下,我们认为训练过程中的深度分配不应是静态预设,而应是一个逐步增长的结构化过程。系统分析揭示了层间由深至浅的成熟轨迹,高熵注意力头在语义整合中起关键作用。受此启发,我们提出稀疏生长Transformer(SGT)。SGT是一种训练时稀疏深度分配框架,通过在信息丰富的注意力头上进行定向循环,逐步从深层向浅层扩展递归。该机制通过仅选择性地增加一小部分参数的深度,诱导结构稀疏性。在多个参数规模上的大量实验表明,与同等设置下的静态块级循环基线相比,SGT始终表现更优,同时将额外训练FLOPs开销从约16%-20%降低至仅1%-3%。

原文摘要 · Abstract (English)

Existing approaches to increasing the effective depth of Transformers predominantly rely on parameter reuse, extending computation through recursive execution. Under this paradigm, the network structure remains static along the training timeline, and additional computational depth is uniformly assigned to entire blocks at the parameter level. This rigidity across training time and parameter space leads to substantial computational redundancy during training. In contrast, we argue that depth allocation during training should not be a static preset, but rather a progressively growing structural process. Our systematic analysis reveals a deep-to-shallow maturation trajectory across layers, where high-entropy attention heads play a crucial role in semantic integration. Motivated by this observation, we introduce the Sparse Growing Transformer (SGT). SGT is a training-time sparse depth allocation framework that progressively extends recurrence from deeper to shallower layers via targeted attention looping on informative heads. This mechanism induces structural sparsity by selectively increasing depth only for a small subset of parameters as training evolves. Extensive experiments across multiple parameter scales demonstrate that SGT consistently outperforms training-time static block-level looping baselines under comparable settings, while reducing the additional training FLOPs overhead from approximately 16--20% to only 1--3% relative to a standard Transformer backbone.

Transformer训练优化稀疏性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。