解释了神经网络为何能连通不同解却难跨过中间区域。
Entropic Confinement and Mode Connectivity in Overparameterized Neural Networks
- 用曲率与优化噪声的相互作用解释熵势垒形成机制。
- 远离极小值点时曲率上升，产生回拉力使优化难以穿越中间路径。
- 揭示曲率诱导的熵力如何决定模型收敛位置和解的连通性。
现代神经网络表现出一个显著特性：损失函数的多个极小值区域之间常存在低损失路径相连，但优化过程通常局限于单一凸谷，极少探索中间区域。我们通过识别由路径上曲率变化与优化噪声相互作用产生的熵势垒，解决了这一悖论。实证发现，远离极小值点时曲率系统性上升，产生有效作用力，将随机优化轨迹拉回端点，即使损失几乎平坦亦如此。此类熵势垒持续时间长于能量势垒，主导参数空间中解的晚期定位。结果表明，曲率诱导的熵力在深度学习景观中同时调控解的连通性与局域化。
原文摘要 · Abstract (English)
Modern neural networks exhibit a striking property: basins of attraction in the loss landscape are often connected by low-loss paths, yet optimization dynamics generally remain confined to a single convex basin and rarely explore intermediate points. We resolve this paradox by identifying entropic barriers arising from the interplay between curvature variations along these paths and noise in optimization dynamics. Empirically, we find that curvature systematically rises away from minima, producing effective forces that bias noisy dynamics back toward the endpoints - even when the loss remains nearly flat. These barriers persist longer than energetic barriers, shaping the late-time localization of solutions in parameter space. Our results highlight the role of curvature-induced entropic forces in governing both connectivity and confinement in deep learning landscapes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。