用傅里叶分形维数预测深度模型泛化能力,无需验证集。
Fourier fractal dimension to predict the generalization of deep neural networks

- 基于权重变化的傅里叶分形维数构建新泛化度量。
- 在CIFAR-10等数据集上与真实泛化差距相关性达顶尖水平。
- 适用于模型稳定性分析与优化算法设计,适合研究者参考。
不依赖预留验证数据预测深度神经网络泛化性能是机器学习中的基本挑战。尽管随机梯度下降(SGD)驱动这些高参数化模型的优化,其重尾、非高斯的动力学在参数空间中引发复杂且尺度不变的轨迹。本文提出一种基于网络权重变化傅里叶分形维数的新泛化度量。通过分析由莱维驱动随机微分方程在频域中的特征函数,提取出能稳健捕捉学习过程几何复杂性的指标。此外,我们设计了一种定制化的傅里叶优化器,可在训练中主动正则化该分形维数。在CIFAR-10、SVHN和MNIST数据集上的大量实证评估表明,所提傅里叶泛化度量与实际泛化差距具有强相关性,其肯德尔等级相关系数优于多种现有基于范数、边界和PAC-Bayesian的度量。该工作揭示了频域分形分析在预测模型泛化能力及构建更稳定优化算法方面的潜力。
原文摘要 · Abstract (English)
Predicting the generalization performance of deep neural networks without relying on hold-out validation data is a fundamental challenge in machine learning. While Stochastic Gradient Descent (SGD) drives the optimization of these highly parameterized models, its heavy-tailed, non-Gaussian dynamics induce complex, scale-invariant trajectories in the parameter space. In this paper, we propose a novel generalization measure based on the Fourier fractal dimension of the network's weight variations. By analyzing the characteristic function of the Lévy-driven stochastic differential equations in the frequency domain, we extract a metric that robustly captures the geometric complexity of the learning process. Furthermore, we introduce a customized Fourier-based optimizer designed to actively regularize this fractal dimension during training. Extensive empirical evaluations on the CIFAR-10, SVHN, and MNIST datasets demonstrate that our proposed Fourier generalization measure exhibits a strong correlation with the actual generalization gap. Our method achieves state-of-the-art Kendall rank correlation coefficients, outperforming a wide array of existing norm-based, margin-based, and PAC-Bayesian measures. Ultimately, this work highlights the potential of frequency-domain fractal analysis as both a powerful predictor for model generalizability and a principled foundation for developing more stable optimization algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。