证明神经网络的坐标下降法能收敛到全局最优解
Block Coordinate Descent for Neural Networks Provably Finds Global Minima
- 采用块坐标下降法优化网络，保证损失指数下降
- 首次理论证明可达到任意小的损失值
- 适合关注优化保证与泛化性能的研究者
本文研究深度神经网络的块坐标下降（BCD）算法，在严格单调递增激活函数下提供新的全局收敛性保证。尽管已有工作证明BCD可收敛至驻点，但本工作首次证明其可收敛至全局最小值，确保损失任意小。我们发现输出层损失呈指数下降，而隐藏层损失保持良好控制。进一步利用Rademacher复杂度框架推导泛化界，表明BCD兼具优异优化性能与良好泛化能力。此外，针对ReLU激活函数（非严格单调），提出含跳跃连接与非负投影的改进算法，将收敛性扩展至该情形。实验验证理论结果：无论在严格单调或ReLU激活下，BCD均能实现小损失。
原文摘要 · Abstract (English)
In this paper, we consider a block coordinate descent (BCD) algorithm for training deep neural networks and provide a new global convergence guarantee under strictly monotonically increasing activation functions. While existing works demonstrate convergence to stationary points for BCD in neural networks, our contribution is the first to prove convergence to global minima, ensuring arbitrarily small loss. We show that the loss with respect to the output layer decreases exponentially while the loss with respect to the hidden layers remains well-controlled. Additionally, we derive generalization bounds using the Rademacher complexity framework, demonstrating that BCD not only achieves strong optimization guarantees but also provides favorable generalization performance. Moreover, we propose a modified BCD algorithm with skip connections and non-negative projection, extending our convergence guarantees to ReLU activation, which are not strictly monotonic. Empirical experiments confirm our theoretical findings, showing that the BCD algorithm achieves a small loss for strictly monotonic and ReLU activations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。