arXiv:2608.24568cs.LGcs.AI2026-08中稿 · KDD

渐进式增长让模型找到更平坦的解,提升泛化能力。

Across the Loss Landscape with Progressive Growth

  • 通过逐步解锁子空间参数,冻结正交方向,模拟约束松弛。
  • 在真实ResNet/CIFAR-100上验证:渐进增长显著降低曲率,提升平坦度。
  • 揭示平坦性与泛化间非线性关系,适合研究优化几何的学者。

深度神经网络在高度非凸、过参数化的损失曲面中仍能良好泛化,这一现象常与随机优化所找到极小值的几何特性相关。本文将参数增长视为渐进约束松弛过程,从低维子模型出发,迭代地解锁嵌套的随机子空间,同时冻结网络初始化时正交于可训练部分的分量,每次扩展后重新优化,直至完整架构。在标准局部正则性条件下,我们证明局部次水平集可被椭球近似,且冻结约束下的基底可达性由冻结方向上的显式有效曲率决定。这解释了偏差机制:渐进增长通过冻结约束引发的体积效应,提升了宽基底的相对权重,抑制了尖锐基底。我们在受控的玩具景观和真实的ResNet/CIFAR-100设置中验证了预测,确认渐进子空间增长可稳定产生更平坦的解,但曲率降低并不总带来测试性能提升,揭示了平坦性与泛化之间复杂的关联。代码已开源。

原文摘要 · Abstract (English)

Deep neural networks generalize well despite their highly nonconvex, overparameterized loss landscapes, a phenomenon often associated with the geometry of the minima found by stochastic optimization. We study how incremental grow-and-optimize strategies bias training toward flatter regions by viewing growth as progressive constraint relaxation. Starting from a low-dimensional submodel, we iteratively expand the trainable parameters by unlocking nested random subspaces while freezing the orthogonal complement at the network initialization, re-optimizing after each expansion until the full architecture is reached. Under standard local regularity conditions around non-degenerate minima, we prove that local sublevel sets are well approximated by ellipsoids and that basin accessibility under frozen constraints can be characterized by an explicit effective curvature in the frozen directions. This leads to an explanation of the bias: progressive growth increases the relative weight of wide basins and suppresses sharp ones through a volume effect induced by the frozen constraints. We empirically validate these predictions in controlled toy landscapes and in a realistic ResNet/CIFAR-100 setting and confirm that although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection. The code is available at https://github.com/p0lcAi/Across-the-Loss-Landscape.

优化几何泛化分析渐进训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。