将深度线性网络训练转化为可精确求解的凸优化问题
Exact Convex Reformulations of Linear Neural Networks via Completely Positive Lifting
- 通过完全正定提升，将多层参数化转化为双线性分解并建模为半定规划
- 最优值与原非凸问题一致，提升维度仅依赖输入输出维数
- 揭示线性网络训练与共轭规划的深层联系，适合理论研究者
我们证明，在平方损失下，深度线性神经网络的训练问题可在提升空间中被精确地转化为广义完全正定锥上的凸问题。该重构问题与原非凸问题具有相同的最优值，且在提升变量上是线性的，所有非凸性均由锥约束编码。其环境提升维度仅取决于输入和输出维度，与网络深度及数据点数量无关，瓶颈宽度仅通过标量约束影响。构造过程包括将多层参数化约化为双线性因子分解，将其提升为秩约束半定规划，用互补条件表达秩约束，并应用完全正定提升。尽管该公式在一般情况下计算不可行，但它给出了线性因子分解引起的非凸性的确切锥表示，并将线性神经网络训练与共轭规划联系起来。
原文摘要 · Abstract (English)
We show that the training problem of a deep linear neural network under the squared loss admits an exact convex reformulation in a lifted space over a generalized completely positive cone. The reformulation has the same optimal value as the original nonconvex problem and is linear in the lifted variables, with all nonconvexity encoded in the cone constraint. Its ambient lifted dimension depends only on the input and output dimensions, independent of the network depth and the number of data points, and the bottleneck width enters only through scalar constraints. The construction proceeds by reducing the multilayer parameterization to a bilinear factorization, lifting it to a rank-constrained semidefinite program, expressing the rank constraint via a complementarity condition, and applying a completely positive lifting. While the resulting formulation is computationally intractable in general, it gives an exact conic representation of the nonconvexity induced by linear factorization and connects linear neural network training with copositive programming.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。