模型在中等数据量时泛化最快，过大反而变慢。
Slower Generalization, Faster Memorization: A Sweet Spot in Algorithmic Learning
- 中等数据量下，小模型泛化速度最快
- 超过临界数据量后，需更多梯度更新才能泛化
- 适合研究模型学习机制与数据规模关系的学者
针对关键数据量的grokking现象，传统观点认为数据足够后继续增加应加速验证收敛。我们发现在结构化输出任务中这一直觉可能失效：在Needleman-Wunsch（NW）矩阵生成任务中，小Transformer在中等数据量时达到最高验证精确匹配准确率所需梯度更新最少，而非最大数据量。越过此数据量甜点后，虽仍可实现泛化，但需更多更新。相反，在部分验证能力初现阶段，更大数据集反而能以更少更新达到高训练准确率，表明规则结构的浮现可加速拟合，超越逐例记忆。乘法基线未呈现相同后期放缓现象。结果揭示了泛化开始所需的最小数据量与最优更新收敛数据量之间的差异，明确了学习规则与完全拟合在结构化输出任务中可分离的情形。
原文摘要 · Abstract (English)
Critical-data-size accounts of grokking suggest a natural post-threshold intuition: once training data is sufficient to identify the underlying rule, additional data should accelerate validation convergence. We show that this intuition can fail in a controlled structured-output task. In Needleman--Wunsch (NW) matrix generation, small Transformers reach high validation exact-match accuracy fastest at an intermediate dataset size, not at the largest one. Past this dataset-size sweet spot, generalization remains achievable but requires more gradient updates. Conversely, in the regime where partial validation competence first appears, larger datasets can require fewer updates to reach high training accuracy, suggesting that emerging rule structure can accelerate fitting beyond example-wise memorization. A multiplication baseline does not show the same post-threshold slowdown. These results separate the critical data size for the onset of generalization from the dataset size that optimizes update-based convergence, and identify structured-output tasks where learning the rule and completing exact-fitting can diverge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。