Adam优化器的偏差修正并非必要,反而可能拖累性能。
Adam Simplified: Bias Correction Debunked
- 通过系统消融实验发现,最优超参下偏差修正无提升
- 不当学习率调度时,偏差修正反而降低模型表现
- 偏差修正本质是隐式学习率调度,依赖平滑参数
Adam优化器是现代深度学习的基石,但其各组件的实际必要性常被默认。本文针对偏差修正这一长期未明的机制展开系统研究。在视觉与语言建模任务上进行系列消融实验表明,常规认知存在误导:在最优超参数配置下,引入偏差修正对最终测试性能无提升;若未采用合适的学习率调度,偏差修正甚至可能损害性能。进一步分析表明,偏差修正实质是一种受平滑参数 $β_1, β_2 o [0,1)$ 影响的隐式学习率调度机制。研究结果挑战了该组件的普遍适用性。
原文摘要 · Abstract (English)
The Adam optimizer is a cornerstone of modern deep learning, yet the empirical necessity of each of its individual components is often taken for granted. This paper presents a focused investigation into the role of bias-correction, a feature whose contribution remains poorly understood. Through a series of systematic ablations on vision and language modelling tasks, we demonstrate that the conventional wisdom surrounding bias correction is misleading. In particular, we demonstrate that in the optimal hyper-parameter configuration, the inclusion of bias correction leads to no improvement in final test performance. Moreover, unless appropriate learning rate scheduling is implemented, the inclusion of bias correction can sometimes be detrimental to performance. We further reinterpret bias correction as a form of implicit learning rate scheduling whose behaviour is strongly dependent on the choice of smoothing hyper-parameters $β_1, β_2 \in [0,1)$. Our findings challenge the universal inclusion of this component.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。