打破数据对称性才能实现特征学习的泛化，关键在恢复数据内在对称性。
Breaking Data Symmetry is Needed For Generalization in Feature Learning Kernels
- 用递归特征机通过平均梯度外积更新特征矩阵。
- 仅当训练集对称性被打破时，模型才出现泛化现象。
- 学出的特征矩阵能编码数据的对称性群结构，适合研究泛化机制者阅读。
当模型达到高训练准确率后，对未见测试点的泛化能力却会延迟出现，这种现象称为Grokking，最初在模运算等代数任务中被观察到。本文研究在特征学习核框架下，使用递归特征机（RFM）算法处理代数任务时的Grokking现象。RFM通过平均梯度外积（AGOP）迭代更新特征矩阵以学习任务相关特征。实验发现，只有当训练集的某种对称性被打破时，泛化才会发生。进一步实证表明，RFM通过恢复数据中隐含的不变性群作用实现泛化，且学习到的特征矩阵编码了该群的具体元素，解释了泛化对对称性的依赖关系。
原文摘要 · Abstract (English)
Grokking occurs when a model achieves high training accuracy but generalization to unseen test points happens long after that. This phenomenon was initially observed on a class of algebraic problems, such as learning modular arithmetic (Power et al., 2022). We study grokking on algebraic tasks in a class of feature learning kernels via the Recursive Feature Machine (RFM) algorithm (Radhakrishnan et al., 2024), which iteratively updates feature matrices through the Average Gradient Outer Product (AGOP) of an estimator in order to learn task-relevant features. Our main experimental finding is that generalization occurs only when a certain symmetry in the training set is broken. Furthermore, we empirically show that RFM generalizes by recovering the underlying invariance group action inherent in the data. We find that the learned feature matrices encode specific elements of the invariance group, explaining the dependence of generalization on symmetry.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。