权重衰减让神经网络权重矩阵变低秩，提升泛化能力
Towards Better Generalization: Weight Decay Induces Low-rank Bias for Neural Networks
- 用随机梯度下降加权重衰减训练时，权重矩阵趋于低秩
- 充分训练后权重矩阵近似为二秩，实验证明此现象普遍
- 无需依赖数据分布假设，适用于各类回归与分类任务
我们研究了在使用权重衰减（WD）训练神经网络（NN）时，其隐含的低秩偏置。理论证明，当使用随机梯度下降（SGD）和权重衰减对一个ReLU神经网络进行充分训练时，其权重矩阵近似为二秩矩阵。实验表明，无论在回归还是分类任务中，权重衰减都是诱导这一低秩偏置的必要条件。与以往研究不同，我们的理论分析不依赖于训练数据分布、权重矩阵最优性或特定训练过程等常见假设。此外，利用该低秩偏置，我们推导出更优的泛化误差界，并通过数值证据表明可实现更好泛化性能。因此，本工作从理论和实证两方面揭示了在结合权重衰减时，SGD表现出强泛化能力的原因。
原文摘要 · Abstract (English)
We study the implicit bias towards low-rank weight matrices when training neural networks (NN) with Weight Decay (WD). We prove that when a ReLU NN is sufficiently trained with Stochastic Gradient Descent (SGD) and WD, its weight matrix is approximately a rank-two matrix. Empirically, we demonstrate that WD is a necessary condition for inducing this low-rank bias across both regression and classification tasks. Our work differs from previous studies as our theoretical analysis does not rely on common assumptions regarding the training data distribution, optimality of weight matrices, or specific training procedures. Furthermore, by leveraging the low-rank bias, we derive improved generalization error bounds and provide numerical evidence showing that better generalization can be achieved. Thus, our work offers both theoretical and empirical insights into the strong generalization performance of SGD when combined with WD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。