通过均衡梯度更新速度，让模型更快实现突然的泛化飞跃。
Egalitarian Gradient Descent: A Simple Approach to Accelerated Grokking
- 用梯度主方向速度均衡化改进优化过程
- 在模加和稀疏奇偶性问题上消除学习停滞期
- 适合关注模型泛化加速与训练动力学的研究者
Grokking 是一种现象：模型的训练性能早期达到峰值，而测试/泛化性能在大量训练周期中持续停滞，随后突然跃升至近乎完美的水平。实践中，缩短这种停滞期极为重要。本文首次从理论和实验两方面证明，梯度下降在不同主方向上的速度不对称是引发 grokking 的原因。为此提出一种简单修改：对梯度进行归一化，使各主方向的更新速度完全一致，称为平等梯度下降（EGD），其可视为自然梯度下降的改良形式。我们证明该方法能显著加速 grokking，甚至彻底消除停滞。在经典算术任务如模加法和稀疏奇偶性问题上，实验验证 EGD 成功消除了长期停滞现象。
原文摘要 · Abstract (English)
Grokking is the phenomenon whereby, unlike the training performance, which peaks early in the training process, the test/generalization performance of a model stagnates over arbitrarily many epochs and then suddenly jumps to usually close to perfect levels. In practice, it is desirable to reduce the length of such plateaus, that is to make the learning process "grok" faster. In this work, we provide new insights into grokking. First, we show both empirically and theoretically that grokking can be induced by asymmetric speeds of (stochastic) gradient descent, along different principal (i.e singular directions) of the gradients. We then propose a simple modification that normalizes the gradients so that dynamics along all the principal directions evolves at exactly the same speed. Then, we establish that this modified method, which we call egalitarian gradient descent (EGD) and can be seen as a carefully modified form of natural gradient descent, groks much faster. In fact, in some cases the stagnation is completely removed. Finally, we empirically show that on classical arithmetic problems such as modular addition and sparse parity problem which this stagnation has been widely observed and intensively studied, that our proposed method eliminates the plateaus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。