arXiv:2606.14259cs.LG2026-06被引 1

Adam比SGD性能更好并非单一原因,而是数据与架构的复杂互动。

Beyond a Single Explanation of the Adam--SGD Gap

  • 通过控制实验检验多种假设,发现无单一因素主导差距
  • 批量大小存在转折点,超过后Adam优势反超SGD
  • 适用于视觉、语言、基因组等多领域模型调参参考

先前研究指出影响Adam与SGD性能差距的因素涵盖数据特性、网络结构和优化属性等多个方面,但这些解释常被孤立看待,重要性不明确。本文在视觉、语言、基因组和图任务上,通过跨现代与经典架构的受控实证研究,系统考察了这些假设。结果表明:无单一因素能始终解释该差距。例如,当词汇分布均匀时Adam优势仍存,而在重尾分布下几乎消失;在softmax注意力模型中,优势甚至反转为SGD更优;在软结构修改(如将ReLU替换为GeLU)下,差距反而增大。这说明差距源于非平凡的数据-架构交互。然而,在各场景中观察到统一模式:存在一个‘交叉批量大小’,当批量增大时,相对优势从SGD转向Adam。这一现象可由我们的理论差距模型准确预测。本研究整合现有假设,并为多领域实践提供新洞见。

原文摘要 · Abstract (English)

Prior work has identified several factors that can contribute to the performance gap between Adam and SGD, spanning data aspects, architecture design, and optimization properties. Yet these explanations are often studied in isolation, leaving their relative importance unclear. In this work, we revisit these hypotheses through a controlled empirical study across vision, language, genomics, and graph tasks, spanning modern and classical architectures, and carefully designed training setups. Our results suggest that no single factor consistently explains the Adam--SGD gap. For instance, the Adam advantage can (1) persist under a uniform vocabulary distribution yet nearly disappear under a heavy-tailed one; (2) reverse in favor of SGD in softmax-attention models; and (3) become larger under soft architectural modifications, e.g., when ReLU is replaced by a GeLU nonlinearity. This suggests that the gap arises from nontrivial data and architecture interactions, rather than from a single common factor. Yet, we observe a pattern across our settings: a \emph{crossover batch size} at which the relative advantage shifts from SGD to Adam as the batch size scales. These empirical results are captured by our theoretical gap model, which predicts this batch-size-dependent crossover. Our perspective helps reconcile several existing hypotheses while offering practical insights across domains.

优化器对比批量大小模型架构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。