研究两层ReLU卷积网络全可训练时的初始化影响，揭示过拟合仍能泛化的关键条件。
Initialization Matters: On the Benign Overfitting of Two-Layer ReLU CNN with Fully Trainable Layers
- 分析全可训练两层ReLU CNN的初始化作用，发现输出层缩放系数决定训练动态
- 大缩放时隐藏层快速增长而输出层几乎不变，小缩放时两层协同增长并保持比例
- 给出测试误差上下界，明确过拟合仍能泛化所需初始化与信噪比条件
良性过拟合指过参数神经网络能完美拟合训练数据且对未见数据仍有良好泛化能力。尽管已有理论研究，但多局限于输出层固定的两层网络，仅训练隐层权重。本文将分析扩展至输出层也全可训练的两层ReLU卷积神经网络，更贴近实际训练场景。结果表明：输出层的初始化缩放至关重要——大缩放使训练行为类似固定输出情形，隐藏层快速生长而输出层基本不变；小缩放则引发复杂层间交互，隐藏层先以特定比例增长，随后两层共同增长并维持该比例。在两种情况下，均给出了测试误差的近似匹配上下界，明确了实现良性过拟合所需的初始化缩放与信噪比（SNR）的严格条件。数值实验验证了理论结果。
原文摘要 · Abstract (English)
Benign overfitting refers to how over-parameterized neural networks can fit training data perfectly and generalize well to unseen data. While this has been widely investigated theoretically, existing works are limited to two-layer networks with fixed output layers, where only the hidden weights are trained. We extend the analysis to two-layer ReLU convolutional neural networks (CNNs) with fully trainable layers, which is closer to the practice. Our results show that the initialization scaling of the output layer is crucial to the training dynamics: large scales make the model training behave similarly to that with the fixed output, the hidden layer grows rapidly while the output layer remains largely unchanged; in contrast, small scales result in more complex layer interactions, the hidden layer initially grows to a specific ratio relative to the output layer, after which both layers jointly grow and maintain that ratio throughout training. Furthermore, in both settings, we provide nearly matching upper and lower bounds on the test errors, identifying the sharp conditions on the initialization scaling and signal-to-noise ratio (SNR) in which the benign overfitting can be achieved or not. Numerical experiments back up the theoretical results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。