用统计物理方法解释深度学习为何有效,提出可评估网络质量的新指标。
SETOL: A Semi-Empirical Theory of (Deep) Learning
- 基于随机矩阵与量子化学理论,推导出深度学习的半经验模型。
- 通过计算权重矩阵谱密度,可估算各层质量,预测模型性能。
- 新指标ERG与已有指标高度一致,适用于小模型和主流大模型。
我们提出一种半经验学习理论(SETOL),用于解释当前最优神经网络(SOTA NN)的优异表现。该理论形式化阐释了重尾自正则化(HTSR)中关键量——重尾幂律层质量度量α与α̂的起源。此前研究已表明,这些度量可在不访问训练或测试数据的情况下预测预训练SOTA模型的测试准确率趋势。本工作运用统计力学、随机矩阵理论及量子化学中的高级方法,推导出理想学习的新数学前提,提出一个等价于执行一步威尔逊精确重整化群(ERG)的新指标。我们在一个简单的三层多层感知机(MLP)上验证了SETOL的假设与预测,结果与理论预期高度吻合。对于SOTA NN模型,仅需计算各层权重矩阵的经验谱密度(ESD),即可通过SETOL公式估计层质量。我们对比分析了HTSR的α与SETOL的ERG,在MLP及SOTA NN上均发现二者表现出惊人的一致性。
原文摘要 · Abstract (English)
We present a SemiEmpirical Theory of Learning (SETOL) that explains the remarkable performance of State-Of-The-Art (SOTA) Neural Networks (NNs). We provide a formal explanation of the origin of the fundamental quantities in the phenomenological theory of Heavy-Tailed Self-Regularization (HTSR): the heavy-tailed power-law layer quality metrics, alpha and alpha-hat. In prior work, these metrics have been shown to predict trends in the test accuracies of pretrained SOTA NN models, importantly, without needing access to either testing or training data. Our SETOL uses techniques from statistical mechanics as well as advanced methods from random matrix theory and quantum chemistry. The derivation suggests new mathematical preconditions for ideal learning, including a new metric, ERG, which is equivalent to applying a single step of the Wilson Exact Renormalization Group. We test the assumptions and predictions of SETOL on a simple 3-layer multilayer perceptron (MLP), demonstrating excellent agreement with the key theoretical assumptions. For SOTA NN models, we show how to estimate the individual layer qualities of a trained NN by simply computing the empirical spectral density (ESD) of the layer weight matrices and plugging this ESD into our SETOL formulas. Notably, we examine the performance of the HTSR alpha and the SETOL ERG layer quality metrics, and find that they align remarkably well, both on our MLP and on SOTA NNs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。