通过分层分离优化，缓解深度网络交叉熵训练的非凸难题。
A Layer Separation Optimization Framework for Cross-Entropy Training in Deep Learning
- 将深层优化问题分解为可管理的子问题，引入隐藏层辅助变量。
- 理论证明新损失函数是原交叉熵的上界，算法具有递减性。
- 实验验证在全连接与卷积网络中优化效果更优，适合复杂模型训练。
本文研究了使用Softmax交叉熵损失的深度学习优化问题。针对深度网络训练中的强非凸性，提出分层分离策略。对全连接和卷积神经网络的交叉熵模型，引入与隐藏层输出相关的辅助变量，构建分层分离模型，将原始深层嵌套优化问题分解为一系列更易处理的子问题。理论分析证明，新分层分离损失函数为原交叉熵损失的上界。同时设计交替最小化算法，并在适当条件下证明其损失函数具有递减性质。数值实验验证了方法的有效性，表明优化行为显著改善，尤其在全连接和卷积神经网络中表现突出。
原文摘要 · Abstract (English)
This paper investigates the deep learning optimization problem with softmax cross-entropy loss. We propose a layer separation strategy to alleviate the strong nonconvexity encountered during training deep networks. For cross-entropy models with fully connected and convolutional neural networks, we introduce auxiliary variables associated with hidden layer outputs and construct corresponding layer separation models, which decompose the original deeply nested optimization problem into a sequence of more manageable subproblems. We also conduct theoretical analyses, proving that the new layer separation loss provides an upper bound for the original cross-entropy loss. Moreover, we design alternating minimization algorithms and prove that, under appropriate conditions, these algorithms exhibit decreasing properties of the loss function. Numerical experiments validate the effectiveness of the proposed methods and indicate improved optimization behavior, especially for fully connected and convolutional neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。