NGD训练出的模型有效维度更高，更不易退化。
NGD converges to less degenerate solutions than SGD
- 用自然梯度下降（NGD）训练模型，提升其有效维度
- 相比SGD，NGD在两种评估方法中均获得更高的学习系数λ
- 适合关注模型泛化能力与复杂度理论的研究者
模型的自由参数数量是衡量其复杂性的直观方式：参数越多，信息容量越大。然而，这并非准确指标——能记忆训练数据的高维模型仍可能表现良好。有效维度旨在更直接地捕捉模型复杂性，仅计算表达模型功能所需的参数量。奇异学习理论（SLT）提出学习系数λ作为更精确的有效维度度量，它通过描述损失函数局部最小值附近参数空间区域体积随损失增长的速度，引入了高阶项信息。我们比较了使用自然梯度下降（NGD）和随机梯度下降（SGD）训练的模型的λ值，发现采用NGD训练的模型在两种方法（海森迹$ ext{Tr}(oldsymbol{H})$ 和局部学习系数估计$ ilde{λ}(w^*)$）下均表现出更高的有效维度。
原文摘要 · Abstract (English)
The number of free parameters, or dimension, of a model is a straightforward way to measure its complexity: a model with more parameters can encode more information. However, this is not an accurate measure of complexity: models capable of memorizing their training data often generalize well despite their high dimension. Effective dimension aims to more directly capture the complexity of a model by counting only the number of parameters required to represent the functionality of the model. Singular learning theory (SLT) proposes the learning coefficient $ λ$ as a more accurate measure of effective dimension. By describing the rate of increase of the volume of the region of parameter space around a local minimum with respect to loss, $ λ$ incorporates information from higher-order terms. We compare $ λ$ of models trained using natural gradient descent (NGD) and stochastic gradient descent (SGD), and find that those trained with NGD consistently have a higher effective dimension for both of our methods: the Hessian trace $ \text{Tr}(\mathbf{H}) $, and the estimate of the local learning coefficient (LLC) $ \hatλ(w^*) $.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。