首次揭示批量大小如何影响Adam的泛化能力，解释为何实际使用中需精细调参。
Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks
- 分析两层过参数化CNN，揭示批量大小对Adam泛化的影响机制。
- 小批量Adam可实现接近零的测试误差，而全批量版本表现差。
- 理论证明Adam有效权重衰减上限低于AdamW，解释其更敏感的调参需求。
Adam是深度学习中广泛使用的自适应梯度方法，其理论研究也备受关注。然而，现有理论大多聚焦于全批量版本，与实际使用的随机变体有本质差异。与SGD不同，即使学习率趋近于零，随机Adam也不会收敛到全批量版本。本文首次理论刻画了批量大小对Adam泛化性能的影响，基于图像数据上的两层过参数化CNN进行分析。结果表明，尽管带合适权重衰减λ的Adam和AdamW均收敛至测试误差较差的解，但它们的小批量变体却能实现接近零的测试误差。进一步证明，Adam的有效权重衰减上限严格小于AdamW，从理论上解释了为何Adam需要更敏感的λ调参。大量实验验证了这些发现，凸显了批量大小和权重衰减在Adam泛化中的关键作用。
原文摘要 · Abstract (English)
Adam is a popular and widely used adaptive gradient method in deep learning, which has also received tremendous focus in theoretical research. However, most existing theoretical work primarily analyzes its full-batch version, which differs fundamentally from the stochastic variant used in practice. Unlike SGD, stochastic Adam does not converge to its full-batch counterpart even with infinitesimal learning rates. We present the first theoretical characterization of how batch size affects Adam's generalization, analyzing two-layer over-parameterized CNNs on image data. Our results reveal that while both Adam and AdamW with proper weight decay $λ$ converge to poor test error solutions, their mini-batch variants can achieve near-zero test error. We further prove Adam has a strictly smaller effective weight decay bound than AdamW, theoretically explaining why Adam requires more sensitive $λ$ tuning. Extensive experiments validate our findings, demonstrating the critical role of batch size and weight decay in Adam's generalization performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。