从渗流理论看丢弃法训练，揭示网络连接断裂的临界现象
Dropout Neural Network Training Viewed from a Percolation Perspective
- 将丢弃法视为随机删边过程，用渗流模型分析网络连通性
- 发现无偏置网络在丢弃率过高时会因路径中断而训练崩溃
- 为理解丢弃法失效机制提供新视角，适合关注正则化原理的研究者
本文从统计物理中的渗流理论出发，研究深度神经网络（NN）使用丢弃法（dropout）训练时是否存在渗流效应。丢弃法通过在每轮训练中随机移除部分连接，利用随机梯度下降（SGD）更新剩余子网络，其随机删边过程与渗流模型高度相似。若丢弃过多连接导致输入与输出间无有效路径，则网络无法进行数据驱动预测。本文构建了模拟丢弃法的新型渗流模型，刻画网络拓扑结构与连通路径之间的关系。理论分析表明，丢弃法确实存在渗流效应。进一步发现，当网络无偏置项时，该渗流效应可能导致训练崩溃；我们还基于启发式论证认为此问题可能普遍存在于含偏置的网络中。
原文摘要 · Abstract (English)
In this work, we investigate the existence and effect of percolation in training deep Neural Networks (NNs) with dropout. Dropout methods are regularisation techniques for training NNs, first introduced by G. Hinton et al. (2012). These methods temporarily remove connections in the NN, randomly at each stage of training, and update the remaining subnetwork with Stochastic Gradient Descent (SGD). The process of removing connections from a network at random is similar to percolation, a paradigm model of statistical physics. If dropout were to remove enough connections such that there is no path between the input and output of the NN, then the NN could not make predictions informed by the data. We study new percolation models that mimic dropout in NNs and characterise the relationship between network topology and this path problem. The theory shows the existence of a percolative effect in dropout. We also show that this percolative effect can cause a breakdown when training NNs without biases with dropout; and we argue heuristically that this breakdown extends to NNs with biases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。