通过噪声鲁棒性等指标，提前预测模型延迟泛化现象
Tracing the Path to Grokking: Embeddings, Dropout, and Network Activation
- 用丢弃率变化曲线评估模型抗噪能力，捕捉泛化转折点
- 测试准确率在训练过程中出现局部峰值，对应延迟泛化阶段
- 适合研究模型泛化机制的科研人员和深度学习工程师
Grokking 指神经网络在训练准确率提升后，测试准确率才显著上升的延迟泛化现象。本文提出多种可实践的预测指标，包括丢弃率下的方差、鲁棒性、嵌入相似性及稀疏性度量。通过在训练过程中随丢弃率变化绘制的丢弃鲁棒性曲线（DRC），可估计模型在推理时对噪声的抗性，从而识别从记忆到泛化的转变。测试准确率在不同训练检查点上的随机丢弃方差，在泛化阶段出现局部最大值。同时，泛化期间无效神经元比例下降，嵌入趋向与初始化无关的双峰分布，且与余弦相似性模式及数据集对称性相关。这些指标为理解 Grokking 的成因与行为提供了深入洞见。
原文摘要 · Abstract (English)
Grokking refers to delayed generalization in which the increase in test accuracy of a neural network occurs appreciably after the improvement in training accuracy This paper introduces several practical metrics including variance under dropout, robustness, embedding similarity, and sparsity measures, that can forecast grokking behavior. Specifically, the resilience of neural networks to noise during inference is estimated from a Dropout Robustness Curve (DRC) obtained from the variation of the accuracy with the dropout rate as the model transitions from memorization to generalization. The variance of the test accuracy under stochastic dropout across training checkpoints further exhibits a local maximum during the grokking. Additionally, the percentage of inactive neurons decreases during generalization, while the embeddings tend to a bimodal distribution independent of initialization that correlates with the observed cosine similarity patterns and dataset symmetries. These metrics additionally provide valuable insight into the origin and behaviour of grokking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。