arXiv:2607.27000cond-mat.dis-nncs.LG2026-07

研究神经网络在有噪声时的解稳定性,发现高温下仍能有效泛化。

On the robustness of noisy solutions in non-convex neural networks

  • 引入有限温度下的重叠间隙性质,揭示噪声如何改变解空间结构
  • 证明允许训练误差后,可访问区域扩展至更高约束密度
  • 数值验证热噪声使复杂难题中仍实现良好泛化,适合机器学习理论研究者

非凸神经网络优化受解空间几何结构强烈影响:稀疏、孤立的点状簇通常难以算法到达,而宽且平坦的区域虽罕见却易于发现。零温情况下,二元感知机通过重叠间隙性质(OGP)形式化了这一现象,限制了在约束密度 α_{OGP} 以上获得零训练误差解的算法能力。本文将此描述推广至有限温度,允许正训练误差并以统计方式惩罚。首先证明,主导零温平衡测度的冻结一阶副本对称性破缺解在任意有限温度下依然存在。进一步基于单模式吉布斯权重在决策边界附近的光滑性,推导出热弛豫是否消除冻结的一般判据。随后将 OGP 构造扩展至有限温度,显示具有有限能量的密集可访问区域可延伸至 α_{OGP}(ε),该阈值随允许训练误差 ε 增大而升高。最后,在教师-学生设置中,我们表明这些宽广的有限能量区域仍保持良好泛化性能。使用有限能量消息传递算法,数值验证热噪声使在恢复教师模型和找到零温解均计算困难的约束密度范围内,仍能实现有效泛化。

原文摘要 · Abstract (English)

Optimization in non-convex neural network models is strongly influenced by the geometry of the solution space: sparse, isolated, point-like clusters are typically algorithmically inaccessible, whereas wide and flat regions can be found efficiently despite being relatively rare. At zero temperature this picture has been formalized in binary perceptrons through the overlap gap property (OGP), which limits algorithmic access to configurations with zero training error above a critical constraint density $α_{\rm OGP}$. Here we extend this description to finite temperature, where a positive training error is allowed and statistically penalized. We first show that the frozen one-step replica-symmetry-breaking solution, dominating the zero temperature equilibrium measure, survives at any finite temperature. We furthermore derive a general criterion, based on the smoothness of the single-pattern Gibbs weight near the decision boundary, that determines when a finite-temperature relaxation of the loss removes freezing. We then extend the OGP construction to finite temperature and show that dense, algorithmically accessible regions of finite-energy configurations persist beyond $α_{\rm OGP}$, up to a threshold $α_{\rm OGP}(ε)$ that grows with the allowed training error $ε$. Finally, in the teacher-student setting, we show that these wide, finite-energy regions still retain good generalization. Using a finite energy message-passing algorithm, we demonstrate numerically that thermal noise enables effective generalization in the regime of constraint densities where both recovering the teacher and finding a zero temperature solution are computationally hard.

神经网络泛化能力热力学

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。