深度残差网络中,丢弃法与随机梯度掩码渐近等价。
Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets
- 在大深度宽网络下,两种方法的参数更新动态趋同。
- 理论证明其极限行为一致,即使训练方式不同。
- 适合研究深度网络泛化机制的学者参考。
Dropout 和随机梯度掩码(RaM)是两种用于提升深度学习性能的训练技术。二者均引入随机性,但方式不同:Dropout 在前向传播中对激活值施加随机掩码,而 RaM 保持前向传播不变,仅在反向传播中掩码梯度。值得注意的是,RaM 引入的参数更新噪声无偏,因此传统解释 Dropout 有效性的机制(如惩罚效应或抑制神经元共适应)不适用于 RaM。本文证明,在大深度与大宽度渐近条件下,对于残差网络(ResNets),这两种方法在完全特征学习区域均收敛至相同的宏观动力学极限。该渐近等价性适用于多种 Dropout 与 RaM 变体,包括随机深度残差网络中的层级丢弃,尽管收敛速度较慢。事实上,我们还发现其中若干变体在渐近意义上会坍缩至同一极限。
原文摘要 · Abstract (English)
Dropout and Random Gradient Masking (RaM) are two training techniques used to improve performance in deep learning. Both techniques inject randomness into the training dynamics, but in significantly different ways: dropout applies random masks to the activations in the forward pass, whereas RaM leaves the forward pass unchanged and instead masks the gradients. In particular, the noise induced by RaM in the parameter updates is unbiased, so standard explanations for the effectiveness of dropout, such as the penalization effect or the prevention of co-adaptation between neurons, do not apply to RaM. In this work, we show that the difference between the two methods disappears for ResNets in the large depth and width asymptotics: in the complete feature learning regime, they both converge to the same large-scale limiting dynamics. This asymptotic equivalence holds for several variants of dropout and RaM, including layerwise dropout as used in stochastic-depth ResNets, albeit at slower quantitative rates. In fact, we also show that several of these variants collapse to the same limit asymptotically.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。