通过损失曲面与海森分析,提出可高效评估神经网络泛化能力的新方法。
Investigating generalization capabilities of neural networks by means of loss landscapes and Hessian analysis
- 基于海森轴改进损失曲面可视化,解决批归一化导致的误判问题。
- 发现多种神经网络存在典型海森谱,可量化评估泛化性能。
- 在ImageNet-1K等数据集上验证,该方法计算高效且预测准确。
本文利用改进的PyTorch库Loss Landscape Analysis(LLA)研究神经网络(NN)的泛化能力。LLA支持损失曲面可视化及海森矩阵性质分析。研究指出,传统可视化方法在含批归一化层的网络中常失效,而采用海森轴可有效缓解此问题,并提出相应的轴选择方法。进一步分析海森特征值谱,发现多种网络具有典型谱分布,据此提出可量化的海森分析标准,用于评估模型性能与泛化能力。实验使用ImageNet-1K预训练模型及本研究训练的多个模型,通过跨数据集训练测试,模拟真实场景下的表现。结果表明,当数据集变化时,所提指标的变化与准确率变化高度相关,证明其为计算高效的泛化能力估计工具,尤其适用于超大规模数据集。
原文摘要 · Abstract (English)
This paper studies generalization capabilities of neural networks (NNs) using new and improved PyTorch library Loss Landscape Analysis (LLA). LLA facilitates visualization and analysis of loss landscapes along with the properties of NN Hessian. Different approaches to NN loss landscape plotting are discussed with particular focus on normalization techniques showing that conventional methods cannot always ensure correct visualization when batch normalization layers are present in NN architecture. The use of Hessian axes is shown to be able to mitigate this effect, and methods for choosing Hessian axes are proposed. In addition, spectra of Hessian eigendecomposition are studied and it is shown that typical spectra exist for a wide range of NNs. This allows to propose quantitative criteria for Hessian analysis that can be applied to evaluate NN performance and assess its generalization capabilities. Generalization experiments are conducted using ImageNet-1K pre-trained models along with several models trained as part of this study. The experiment include training models on one dataset and testing on another one to maximize experiment similarity to model performance in the Wild. It is shown that when datasets change, the changes in criteria correlate with the changes in accuracy, making the proposed criteria a computationally efficient estimate of generalization ability, which is especially useful for extremely large datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。