改进了随机失活的调度策略,显著降低模型测试损失。
Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos
- 将随机失活视为混沌边缘信号传播的扰动,建立均场理论。
- 前段加载的失活策略使测试损失降低18%–35%,优于固定失活。
- 适用于MLP与视觉变压器,为模型正则化提供新思路。
我们建立了随机失活的均场理论,将其视为混沌边缘临界信号传播的扰动。理论表明,一种无需成本的简单改变——前段加载的失活调度——在固定预算下可使MLP和视觉变压器的测试损失降低18%至35%。其机制在于:失活改变了完美对齐不动点,使得信息传播深度在临界初始化下仍为有限值。我们推导出相关性衰减的临界与交叉尺度律,发现平滑激活函数与具有折点的ReLU型激活函数属于不同普适类,具有不同的临界指数,并在失谐与失活强度上实现通用双参数缩放坍缩。这一差异源于相关性映射的解析结构:平滑激活函数在完美对齐附近存在泰勒展开,而折点激活函数则产生具有普适非解析性的分支点。作为推论,该框架在固定预算下得到饱和失活配置;通过正则化范围论证,优选前段加载调度,准确率提升为一致的次要效应。此外,相同的高斯核结构可将理论扩展至卷积神经网络与残差架构。
原文摘要 · Abstract (English)
We develop a mean-field theory of dropout as a perturbation of critical signal propagation at the edge of chaos, and show that it predicts a simple, no-cost change to standard practice: \emph{front-loaded} dropout schedules cut test loss by \(18\)--\(35\%\) over constant dropout in MLPs and Vision Transformers at fixed budget. The theoretical mechanism is that dropout shifts the perfect-alignment fixed point, making the depth scale for information propagation finite even at critical initialization. We derive critical and crossover scaling laws for correlation decay and establish that smooth activations and kinked, \relu{}-like activations constitute distinct universality classes, with different critical exponents and a universal two-parameter scaling collapse in detuning and dropout strength. The distinction traces to the analytic structure of the correlation map: smooth activations admit a Taylor expansion near perfect alignment, while kinked activations develop a branch point with universal non-analyticity. As a corollary, the framework yields saturated dropout profiles under fixed budget; a regularization-reach argument then selects front-loaded schedules, with accuracy gains as a consistent secondary effect. We also discuss how the same Gaussian-kernel structure extends the theory beyond MLPs toward CNNs and residual architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。