arXiv:2411.09807cs.LGcs.AI2024-11被引 5

用拓扑分析方法量化神经网络损失曲面形状,揭示模型性能新规律。

Evaluating Loss Landscapes from a Topology Perspective

  • 引入拓扑数据分析技术,量化损失曲面的几何结构
  • 发现不同模型的损失曲面拓扑特征与性能指标相关
  • 适合研究模型优化机制和训练动态的科研人员

从参数角度刻画神经网络的损失曲面(loss landscape)可为模型性质提供重要洞见。尽管已有多种可视化方法,但对这些复杂表示进行量化并提取可行动、可复现的结论仍较少。受拓扑数据分析(TDA)工具启发,本文从拓扑角度表征损失曲面的底层结构,实现量化分析以揭示新见解。为关联机器学习文献,我们计算准确率、误差等基础性能指标,并利用海森矩阵相关度量(如最大特征值、迹、特征值谱密度)刻画局部结构。基于该方法,我们研究了图像识别中的经典模型(如ResNets)和科学机器学习中的模型(如物理信息神经网络),表明量化损失曲面的拓扑结构能为模型性能与学习动态提供新认知。

原文摘要 · Abstract (English)

Characterizing the loss of a neural network with respect to model parameters, i.e., the loss landscape, can provide valuable insights into properties of that model. Various methods for visualizing loss landscapes have been proposed, but less emphasis has been placed on quantifying and extracting actionable and reproducible insights from these complex representations. Inspired by powerful tools from topological data analysis (TDA) for summarizing the structure of high-dimensional data, here we characterize the underlying shape (or topology) of loss landscapes, quantifying the topology to reveal new insights about neural networks. To relate our findings to the machine learning (ML) literature, we compute simple performance metrics (e.g., accuracy, error), and we characterize the local structure of loss landscapes using Hessian-based metrics (e.g., largest eigenvalue, trace, eigenvalue spectral density). Following this approach, we study established models from image pattern recognition (e.g., ResNets) and scientific ML (e.g., physics-informed neural networks), and we show how quantifying the shape of loss landscapes can provide new insights into model performance and learning dynamics.

拓扑分析损失曲面神经网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。