揭示神经网络损失曲面中模式连接的机制，助力多样化特征学习。
From the Loss Landscape to Diverse Feature Learning in Neural Networks
- 通过分析损失曲面结构，解释模式连接现象
- 发现不同模型间存在平滑路径，支持多样化特征学习
- 为理解训练过程与模型泛化提供新视角，适合研究者参考
过去十年，神经网络已从学术兴趣发展为推动国家市场的关键力量。尽管研究与应用迅猛增长，我们对它们如何达成解决方案的理解仍十分有限。这不仅具有科学意义，更关乎社会影响——当神经网络在自动驾驶、建筑、法律、招聘和医疗等领域做出决策时，不可避免地会产生意外后果。然而，要概括最大生产系统中的失败模式极为困难。实际上，这些失败信号存在于所有规模的神经网络中，因此研究更易处理的设置更具可行性。所有神经网络都需经历训练这一优化过程才能发挥作用。在很大程度上，理解神经网络就是理解其优化过程：通过何种途径、在哪些数据下抵达当前结果。但该领域的知识仍相当模糊。特别是‘模式连接’这一奇特现象——即能在损失曲面上连接不同神经网络——至今无法解释。本论文阐明、解释并利用了损失曲面中的这一特殊结构。
原文摘要 · Abstract (English)
Over the course of the last decade, neural networks have grown from an academic curiosity to moving the markets of nations. Despite this explosion in both research and deployment, relatively little is understood about how they achieve the solutions they do. This is both scientifically relevant, and pressing for society. When neural networks make decisions across self-driving, construction, law, hiring and health, there have been and will continue to be unintended consequences. However, attempting to generalize the failures of the largest and most important production systems makes for a very difficult task. Yet signs of these failures exist at all scales of neural networks, so we should be able to study a much more tractable setting. All neural networks must undergo an optimization process, called training, to be useful. To a great degree, understanding neural networks is understanding their optimization: through what process and exposure to which data did they arrive at their results. Yet our knowledge on this topic as a field is quite imprecise. In particular, a curious phenomenon called mode connectivity, the ability to connect neural networks in the loss surface, defies explanation entirely. This dissertation elucidates, explains and exploits this special structure in the loss landscape...
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。