用梯度下降的隐式正则化提升神经网络泛化能力
Variational Deep Learning via Implicit Regularization
- 仅依赖梯度下降的隐式偏置来正则化变分神经网络
- 在分布内和分布外均表现良好，无需额外调参
- 计算开销极小，适合实际部署
现代深度学习模型在过参数化且缺乏显式正则化的情况下仍表现出优异的分布内泛化能力。当前理论认为这是由架构、超参数和优化过程带来的隐式正则化所致。然而，深层神经网络往往对扰动敏感，导致过度自信预测和分布外泛化性能差。贝叶斯深度学习通过模型平均缓解此问题，但通常需要大量计算资源及精心设计的先验，可能抵消隐式正则化的优点。本文提出仅依赖（随机）梯度下降的隐式偏差来正则化变分神经网络。我们理论上将过参数化线性模型中的这种归纳偏置表征为广义变分推断，并揭示参数化选择的重要性。实验表明，该方法在无需额外超参数调优且计算开销极低的情况下，实现了出色的分布内与分布外性能。
原文摘要 · Abstract (English)
Modern deep learning models generalize remarkably well in-distribution, despite being overparametrized and trained with little to no explicit regularization. Instead, current theory credits implicit regularization imposed by the choice of architecture, hyperparameters, and optimization procedure. However, deep neural networks can be surprisingly non-robust, resulting in overconfident predictions and poor out-of-distribution generalization. Bayesian deep learning addresses this via model averaging, but typically requires significant computational resources as well as carefully elicited priors to avoid overriding the benefits of implicit regularization. Instead, in this work, we propose to regularize variational neural networks solely by relying on the implicit bias of (stochastic) gradient descent. We theoretically characterize this inductive bias in overparametrized linear models as generalized variational inference and demonstrate the importance of the choice of parametrization. Empirically, our approach demonstrates strong in- and out-of-distribution performance without additional hyperparameter tuning and with minimal computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。