arXiv:2510.26303cs.LGcs.AI2025-10被引 2

增量Adam在可分数据上会偏离全批量行为,可能收敛到$oldsymbol{ ext{ℓ}_2}$最大间隔解。

Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regime

  • 研究单样本更新的Adam在可分数据上的隐式偏差
  • 在特定数据集上证明其收敛至ℓ₂最大间隔分类器
  • 揭示批次大小和数据分布对优化器选择的影响

Adam是深度学习中默认使用的优化器,但其理论理解仍不充分。已有分析表明,Adam倾向于与ℓ∞-几何对齐的解,但这些结果仅限于全批量情形。本文研究增量Adam(每步使用一个样本)在线性可分数据上的隐式偏差,发现其行为可偏离全批量情况。极端情况下,我们构造的数据集使增量Adam严格收敛于ℓ₂-最大间隔分类器,而全批量Adam则呈现ℓ∞-最大间隔偏差。对于一般数据,我们通过β₂→1极限的代理算法刻画其偏差:该算法最大化一个由数据自适应马哈拉诺比范数定义的间隔,其协方差矩阵由数据相关的对偶不动点决定。我们进一步给出具体实例,说明此偏差退化为标准ℓ₂或ℓ∞-最大间隔分类器。作为对比,我们证明Signum在任意批量下均收敛至ℓ∞-最大间隔分类器。总体而言,我们的结果表明Adam的隐式偏差高度依赖于批处理方式和数据分布,而Signum则保持不变。

原文摘要 · Abstract (English)

Adam [Kingma & Ba, 2015] is the de facto optimizer in deep learning, yet its theoretical understanding remains limited. Prior analyses show that Adam favors solutions aligned with $\ell_\infty$-geometry, but these results are restricted to the full-batch regime. In this work, we study the implicit bias of incremental Adam (using one sample per step) for logistic regression on linearly separable data, and show that its bias can deviate from the full-batch behavior. As an extreme example, we construct datasets on which incremental Adam provably converges to the $\ell_2$-max-margin classifier, in contrast to the $\ell_\infty$-max-margin bias of full-batch Adam. For general datasets, we characterize its bias using a proxy algorithm for the $β_2 \to 1$ limit. This proxy maximizes a data-adaptive Mahalanobis-norm margin, whose associated covariance matrix is determined by a data-dependent dual fixed-point formulation. We further present concrete datasets where this bias reduces to the standard $\ell_2$- and $\ell_\infty$-max-margin classifiers. As a counterpoint, we prove that Signum [Bernstein et al., 2018] converges to the $\ell_\infty$-max-margin classifier for any batch size. Overall, our results highlight that the implicit bias of Adam crucially depends on both the batching scheme and the dataset, while Signum remains invariant.

优化器隐式偏差Adam最大间隔

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。