arXiv:2605.06169cs.LGcs.CV2026-05

解决1000层扩散模型深度坍缩问题,提出分离均值与方差的残差结构。

Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformers

论文配图:Mean Mode Screaming: Mean--Variance Split Residuals for 1000-Layer Diffusion Transformers
图 1 · 摘自论文原文
  • 将残差更新分解为均值与中心化两部分,防止均值主导网络
  • 在400层模型上稳定训练,避免崩溃且优于层归一化方法
  • 成功实现1000层深度可训练,验证架构极限扩展能力

将扩散Transformer(DiT)扩展至数百层时,网络易进入静默的均值主导坍缩状态,导致令牌表征同质化并抑制中心变化。通过机制审计,我们识别出该坍缩的触发机制为均值模式尖叫(MMS)。即使训练看似稳定,均值相干的反向冲击仍会激活深层残差分支,使网络陷入均值主导状态。此行为源于梯度被精确分解为均值相干与中心分量,且当数值同质化后,注意力对数梯度在Softmax雅可比零空间中被结构抑制。为此,我们提出均值-方差分离(MV-Split)残差:结合独立获得的中心残差更新与泄漏式主干均值替换。在400层单流DiT上,MV-Split阻止了基线的发散崩溃,轨迹保持接近基线预崩溃阶段,且显著优于如LayerScale等令牌各向同性门控方法。最后,我们展示了一例1000层DiT作为边界尺度的规模验证,证明该架构可在极端深度下稳定训练。

原文摘要 · Abstract (English)

Scaling Diffusion Transformers (DiTs) to hundreds of layers introduces a structural vulnerability: networks can enter a silent, mean-dominated collapse state that homogenizes token representations and suppresses centered variation. Through mechanistic auditing, we isolate the trigger event of this collapse as Mean Mode Screaming (MMS). MMS can occur even when training appears stable, with a mean-coherent backward shock on residual writers that opens deep residual branches and drives the network into a mean-dominated state. We show this behavior is driven by an exact decomposition of these gradients into mean-coherent and centered components, compounded by the structural suppression of attention-logit gradients through the null space of the Softmax Jacobian once values homogenize. To address this, we propose Mean-Variance Split (MV-Split) Residuals, which combine a separately gained centered residual update with a leaky trunk-mean replacement. On a 400-layer single-stream DiT, MV-Split prevents the divergent collapse that crashes the un-stabilized baseline; it tracks close to the baseline's pre-crash trajectory while remaining substantially better than token-isotropic gating methods such as LayerScale across the full schedule. Finally, we present a 1000-layer DiT as a scale-validation run at boundary scales, establishing that the architecture remains stably trainable at extreme depth.

扩散模型深度网络残差结构可扩展性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。