揭示大模型中丢弃法的五种渐近相变,发现其本质是随机几何更新。
Phase Diagram of Dropout for Two-Layer Neural Networks in the Mean-Field Regime
- 通过均场极限分析两层网络梯度下降中的丢弃法,构建五相图。
- 大学习率下丢弃不再起惩罚作用,等价于后向传播后的随机梯度稀释。
- 适用于理解大规模神经网络中丢弃法的理论机制,适合研究者阅读。
Dropout 是一种标准的神经网络训练技术,通过在每步梯度更新中随机关闭单元来提升性能,尤其在语言或视觉大模型训练中效果显著。为理解其在大模型中的作用,本文研究了在均场初始化尺度下,两层神经网络梯度下降与丢弃法的大宽度渐近行为。我们得到一个丰富的渐近相图,根据丢弃率、学习率和网络宽度的相对大小,呈现出五个非退化的不同相。值得注意的是,传统认为的“惩罚”效应仅在学习率极小($O(1/ ext{width})$)时成立;对于更大的学习率,该效应消失,丢弃法在极限下等价于一种“随机几何”技术:在前向与反向传播完成后,随机稀释梯度。此渐近态由均场跳跃过程描述,神经元的更新时间遵循独立泊松或伯努利钟(取决于学习率是否趋于零)。部分相中,我们同时给出了路径空间与分布空间的极限动力学描述。收敛性证明结合了均场粒子系统与随机过程工具。本工作为重新理解大模型中丢弃法的理论基础提供了关键支撑。
原文摘要 · Abstract (English)
Dropout is a standard training technique for neural networks that consists of randomly deactivating units at each step of their gradient-based training. It is known to improve performance in many settings, including in the large-scale training of language or vision models. As a first step towards understanding the role of dropout in large neural networks, we study the large-width asymptotics of gradient descent with dropout on two-layer neural networks with the mean-field initialization scale. We obtain a rich asymptotic phase diagram that exhibits five distinct nondegenerate phases depending on the relative magnitudes of the dropout rate, the learning rate, and the width. Notably, we find that the well-studied "penalty" effect of dropout only persists in the limit with impractically small learning rates of order $O(1/\text{width})$. For larger learning rates, this effect disappears and in the limit, dropout is equivalent to a "random geometry" technique, where the gradients are thinned randomly after the forward and backward pass have been computed. In this asymptotic regime, the limit is described by a mean-field jump process where the neurons' update times follow independent Poisson or Bernoulli clocks (depending on whether the learning rate vanishes or not). For some of the phases, we obtain a description of the limit dynamics both in path-space and in distribution-space. The convergence proofs involve a mix of tools from mean-field particle systems and stochastic processes. Together, our results lay the groundwork for a renewed theoretical understanding of dropout in large-scale neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。