当神经网络表达能力退化为代数结构时，能否拟合任务决定成败，不再有缓慢记忆或突然泛化的中间态。
Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations
- 用解析激活函数构建可精确求解模型，揭示网络输出受限于特定代数结构
- 非可表示任务即使在训练集上也无法拟合，训练损失有正下界
- 99.8%预测准确率，呈现即时成功或彻底失败的二元结果，无中间态
在模运算任务上训练的两层神经网络采用解析单项式激活函数σ(z)=z^k，其输出被约束在(Z_p)^2特征空间的一个(k+1)维子空间中，仅占全函数空间的O(k/p²)。我们给出该子空间的完整代数刻画：任务可表示当且仅当其离散傅里叶支持位于u+v=k(mod p)，对线性相位目标简化为m+n=k。此限制不仅影响泛化，也制约记忆：不可表示任务无法在训练集上拟合，训练损失存在与宽度无关的正下界。585次实验中，代数预测与实际结果匹配率达99.8%，无记忆阶段也无突现泛化；结果清晰分为即时成功和彻底失败。此二元行为是容量-突现关系的极限情形：当表达类退化为固定代数对象时，是否能突现的问题转化为能否表示目标本身。瓶颈消融分析将此极端与标准网络联系起来，展示从表示失败、记忆但不泛化，到突现并伴随泛化差距缩小的连续路径。
原文摘要 · Abstract (English)
Neural networks trained on modular arithmetic exhibit grokking, a delayed transition from memorisation to generalisation known to depend on model capacity: too little and the network memorises slowly or not at all, too much and it generalises almost immediately. What happens at the extreme of this spectrum, when the architecture's expressible function class collapses to a finite-dimensional algebraic variety? We study two-layer networks with a holomorphic monomial activation sigma(z)=z^k, trained on modular tasks encoded via roots of unity. Here the network output, regardless of hidden width, is confined to a (k+1)-dimensional subspace of characters of (Z_p)^2, an O(k/p^2) slice of the full function space. We give a complete algebraic characterisation of this subspace: a task is representable if and only if its discrete Fourier support lies on the diagonal u+v = k (mod p), which for linear-phase targets reduces to the arithmetic criterion m+n=k. This is not merely a constraint on eventual generalisation but on memorisation itself: because the outputs are algebraically confined, a non-representable target cannot be fit even on the training set, and we prove a positive lower bound on the training loss, independent of width. Across 585 runs the algebraic prediction matches the observed outcome with 99.8% accuracy, with no memorisation regime and no grokking; outcomes split cleanly into instant success and outright failure. This binary behaviour is the limiting case of the capacity-grokking relationship: when the expressible class shrinks to a fixed algebraic object, the question of when a network will grok dissolves into whether it can represent the target at all. A bottleneck ablation connects this extreme to standard networks, tracing a continuous path from representational failure, through memorisation without generalisation, to grokking with a shrinking gap as capacity grows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。