揭示对角线网络训练轨迹与套索正则化路径的深层关联
Diagonal Linear Networks and the Lasso Regularization Path
- 用对角线结构解析梯度下降的隐式正则化机制
- 训练时间等价于逆正则化参数，精确对应套索路径
- 理论+仿真验证，适合研究优化与正则化的读者
对角线线性网络是激活函数为线性且权重矩阵为对角的神经网络。其理论价值在于：从微小初始化出发，训练过程收敛到最小1-范数的训练损失极小化解。本文进一步分析表明，对角线线性网络的完整训练轨迹与套索（Lasso）正则化路径密切相关。训练时间扮演了逆正则化参数的角色。我们提供了严格结果和模拟实验来支持这一结论。在套索路径单调的假设下，二者关系是精确的；一般情况下，我们证明了近似等价关系。
原文摘要 · Abstract (English)
Diagonal linear networks are neural networks with linear activation and diagonal weight matrices. Their theoretical interest is that their implicit regularization can be rigorously analyzed: from a small initialization, the training of diagonal linear networks converges to the linear predictor with minimal 1-norm among minimizers of the training loss. In this paper, we deepen this analysis showing that the full training trajectory of diagonal linear networks is closely related to the lasso regularization path. In this connection, the training time plays the role of an inverse regularization parameter. Both rigorous results and simulations are provided to illustrate this conclusion. Under a monotonicity assumption on the lasso regularization path, the connection is exact while in the general case, we show an approximate connection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。