arXiv:2602.08302cs.LGcs.AI2026-02

线性模型也能出现延迟泛化,源于梯度下降的隐式偏置与数据不对称性。

Grokking in Linear Models for Logistic Regression

  • 在简单线性分类中,通过分析梯度下降的三阶段动态揭示延迟泛化机制。
  • 当测试数据集中在边界或为对抗样本时,模型会出现显著延迟泛化现象。
  • 无需深度网络,数据分布不均和支撑向量不对称即可引发该现象,适合研究泛化动力学者阅读。

Grokking(延迟泛化)通常被认为依赖于深度神经网络的深度与组合结构。本文研究最简单的设置:在原点处线性(最大间隔)可分的数据上,用梯度下降学习线性模型进行二分类,损失函数为逻辑损失。我们考察三种测试情形:(1) 测试数据来自与训练数据相同分布,此时不观察到grokking;(2) 测试数据集中于边界区域,此时观察到grokking;(3) 对抗测试数据通过投影梯度下降(PGD)生成,同样观察到grokking。理论上证明梯度下降的隐式偏置导致三阶段学习过程——总体主导、支持向量主导的遗忘、支持向量主导的泛化——在此过程中可产生延迟泛化。分析进一步表明,grokking的出现与数据不对称性相关,包括类别样本数量差异以及各类别支撑向量分布差异,并给出了grokking时间的刻画。实验通过人为设置不同分布的总体点与支撑向量,验证了理论预测,分析了准确率曲线与超平面演化。结果表明,grokking无需深度或表示学习,即使在线性模型中,仅通过偏置项的动力学也可出现。

原文摘要 · Abstract (English)

Grokking, the phenomenon of delayed generalization, is often attributed to the depth and compositional structure of deep neural networks. We study grokking in one of the simplest possible settings: the learning of a linear model with logistic loss for binary classification on data that are linearly (and max margin) separable about the origin. We investigate three testing regimes: (1) test data drawn from the same distribution as the training data, in which case grokking is not observed; (2) test data concentrated around the margin, in which case grokking is observed; and (3) adversarial test data generated via projected gradient descent (PGD) attacks, in which case grokking is also observed. We theoretically show that the implicit bias of gradient descent induces a three-phase learning process-population-dominated, support-vector-dominated unlearning, and support-vector-dominated generalization-during which delayed generalization can arise. Our analysis further relates the emergence of grokking to asymmetries in the data, both in the number of examples per class and in the distribution of support vectors across classes, and yields a characterization of the grokking time. We experimentally validate our theory by planting different distributions of population points and support vectors, and by analyzing accuracy curves and hyperplane dynamics. Overall, our results demonstrate that grokking does not require depth or representation learning, and can emerge even in linear models through the dynamics of the bias term.

线性模型延迟泛化梯度下降泛化机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。