arXiv:2603.25009cs.LG2026-03被引 6

揭示神经网络延迟泛化的核心机制:优化与正则化协同决定,而非架构本身。

A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization

  • 控制变量实验,分离架构、优化和正则化影响。
  • 深度非单调:4层MLP无法泛化,8层残差网可恢复。
  • 权重衰减是关键调控参数,存在狭窄最优区间。

神经网络中从记忆到泛化的延迟过渡(即Grokking)机制仍不明确,部分原因在于以往研究混淆了架构、优化和正则化的作用。本文在模97加法任务上开展受控实验,对不同模型采用匹配且精细调优的训练设置,系统解耦各因素影响。核心发现为:Grokking动态并非主要由架构决定,而是优化稳定性与正则化相互作用的结果。具体表现为:(1) 深度呈现非单调效应,4层MLP始终无法Grokking,而8层残差网络可恢复泛化,表明深度需架构稳定;(2) 在匹配超参下,Transformer与MLP的差异几乎消失(延迟仅1.11倍),说明此前观察到的差距主要源于优化器和正则化混杂;(3) 激活函数效果依赖训练制度,当正则化允许记忆时,GELU比ReLU快达4.3倍;(4) 权重衰减是主导控制参数,存在狭窄的“恰到好处”区间,过小或过大均会阻止泛化。每组配置3–5个随机种子,结果统一支持Grokking为交互驱动现象。研究挑战了以架构为中心的解释,澄清了优化与正则化共同主导延迟泛化的机制。

原文摘要 · Abstract (English)

Grokking the delayed transition from memorization to generalization in neural networks remains poorly understood, in part because prior empirical studies confound the roles of architecture, optimization, and regularization. We present a controlled study that systematically disentangles these factors on modular addition (mod 97), with matched and carefully tuned training regimes across models. Our central finding is that grokking dynamics are not primarily determined by architecture, but by interactions between optimization stability and regularization. Specifically, we show: (1) \textbf{depth has a non-monotonic effect}, with depth-4 MLPs consistently failing to grok while depth-8 residual networks recover generalization, demonstrating that depth requires architectural stabilization; (2) \textbf{the apparent gap between Transformers and MLPs largely disappears} (1.11$\times$ delay) under matched hyperparameters, indicating that previously reported differences are largely due to optimizer and regularization confounds; (3) \textbf{activation function effects are regime-dependent}, with GELU up to 4.3$\times$ faster than ReLU only when regularization permits memorization; and (4) \textbf{weight decay is the dominant control parameter}, exhibiting a narrow ``Goldilocks'' regime in which grokking occurs, while too little or too much prevents generalization. Across 3--5 seeds per configuration, these results provide a unified empirical account of grokking as an interaction-driven phenomenon. Our findings challenge architecture-centric interpretations and clarify how optimization and regularization jointly govern delayed generalization.

Grokking深度学习正则化优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。