从概率分布角度揭示深度学习优化机制，解释为何过参数化有效。
Towards Understanding the Optimization Mechanisms in Deep Learning
- 用Fenchel-Young损失，同时最小化梯度范数和结构误差
- 过参数化与独立初始化可控制结构误差，逼近全局最优
- 为梯度下降、过参数化提供理论支持，适合研究者参考
本文从概率分布估计视角，研究监督分类中深度神经网络的优化机制。我们证明，在使用Fenchel-Young损失时，尽管模型参数的拟合误差非凸，但通过同时最小化梯度范数和结构误差，仍可逼近全局最优解。梯度范数可通过梯度下降控制；结构误差则可通过增加参数数量并保证参数独立性来管理，从而为过参数化和随机初始化等现象提供理论解释。论文通过实验证实了该方法的关键结论，展示了其实际有效性。
原文摘要 · Abstract (English)
In this paper, we adopt a probability distribution estimation perspective to explore the optimization mechanisms of supervised classification using deep neural networks. We demonstrate that, when employing the Fenchel-Young loss, despite the non-convex nature of the fitting error with respect to the model's parameters, global optimal solutions can be approximated by simultaneously minimizing both the gradient norm and the structural error. The former can be controlled through gradient descent algorithms. For the latter, we prove that it can be managed by increasing the number of parameters and ensuring parameter independence, thereby providing theoretical insights into mechanisms such as over-parameterization and random initialization. Ultimately, the paper validates the key conclusions of the proposed method through empirical results, illustrating its practical effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。