通过梯度流入不对称性,让深层模型变窄或剪枝,提升效率且不损失性能。
CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry

- 深度模型按梯度流入差异分层设计,深层变窄以匹配信息流动不均
- 相同训练预算下,困惑度相当,延迟降8.6%,吞吐量升9.4%
- 无需后验分析即可剪枝,适合追求高效部署的工程师
深度Transformer由均匀堆叠的残差块组成,但最深层常贡献有限。本文提出两种效率方法:CascadeFormer通过随深度递减宽度来匹配层间信息流不均,在相同训练预算下实现与均匀基线相当的困惑度,同时降低8.6%延迟,提升9.4%吞吐量;CascadeFlow Pruning利用累积训练梯度自动剪枝,无需后验分析,优于常规启发式方法在困惑度与排名稳定性上的表现,下游任务准确率也保持竞争力。为支持这些方法,本文提出梯度流入不对称性(GFA)作为结构解释——在预归一化残差结构中,某层梯度是恒等路径与所有下游功能路径之和,导致梯度流入随深度线性衰减(深监督下呈二次衰减),使早期层梯度更丰富、后期层更稀疏。我们在从零训练至12亿参数的模型上提供了相关性和干预性证据。跨Transformer与ResNet,累积训练梯度符合理论流入规律,并与后验层重要性相关。两项干预表明结构而非梯度幅值是瓶颈:均衡各层梯度范数无法恢复深层价值,而通过参数共享重复增加下游路径数量则可恢复并提升其价值。梯度幅值是否在高秩以外仍能代理流入,以及百亿级以上动态如何演变,仍是开放问题。
原文摘要 · Abstract (English)
Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers often add little value. We present two efficiency methods that exploit this asymmetry. CascadeFormer tapers width with depth to match the uneven information flow across layers, achieving comparable perplexity to a uniform baseline at the same training budget while reducing latency by 8.6% and increasing throughput by 9.4%. CascadeFlow Pruning removes layers using accumulated training gradients, with no post hoc analysis. It outperforms standard heuristics on perplexity and rank-stability and stays competitive on downstream accuracy. To motivate these methods, we propose Gradient Fan-in Asymmetry (GFA) as a structural account of why deeper layers contribute less. In Pre-LayerNorm residual stacks, the gradient at a layer is the sum of an identity path and all downstream functional paths, producing a gradient fan-in that decays linearly with depth (and quadratically under deep supervision), yielding richer gradients for early layers and sparser ones for later layers. We provide correlational and interventional evidence for GFA on models trained from scratch up to 1.2B parameters. Across Transformers and ResNets, accumulated training gradients follow the theoretical fan-in and are associated with post hoc layer importance. Two interventions point to structure rather than magnitude as the bottleneck: equalizing per-layer gradient norms does not restore late-layer value, while increasing downstream path counts via parameter-shared repetition restores and elevates it. Whether gradient magnitude proxies fan-in beyond high-rank regimes, and how these dynamics behave at the 100B+ scale, remain open questions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。