新网络通过局部优化实现零误差训练与完美泛化,无需全局损失函数。
Generalization emerges from local optimization in a self-organized learning network
- 用局部规则重构网络拓扑,知识以查表形式存于节点中。
- 数据量足够时,模型在训练后达到100%泛化准确率。
- 泛化突现如一级相变,训练数据与结构形成可分离处理。
我们设计并分析了一种新型监督学习网络范式,仅依赖局部优化规则,不依赖全局误差函数。传统固定拓扑神经网络通过调整连接权重获得表达能力,而我们的网络将新知识精确且即时地存储在节点中,形式为查找表。随后部分信息被结构化并融入网络几何。训练误差从构建起即为零,并在整个拓扑变换阶段保持为零。该阶段涉及少量局部拓扑操作,如节点分裂或合并,以及二元连接的添加。操作选择仅由局部尺度上的表达力优化驱动。我们关注学习网络的核心能力——泛化,即对未学过问题的正确回答能力。在多个分类任务中,当学习样本数足够大时,该算法生成的网络系统性地达到完美泛化状态。我们报告了状态变化的动力学,发现其具有突发性,表现出一级相变特征,这一现象已在传统学习网络中观察到,称为grokking。除了提出一种非势能型学习网络构建方法,我们的算法使grokking过渡得以重新理解:训练数据获取与网络拓扑结构化完全解耦。
原文摘要 · Abstract (English)
We design and analyze a new paradigm for building supervised learning networks, driven only by local optimization rules without relying on a global error function. Traditional neural networks with a fixed topology are made up of identical nodes and derive their expressiveness from an appropriate adjustment of connection weights. In contrast, our network stores new knowledge in the nodes accurately and instantaneously, in the form of a lookup table. Only then is some of this information structured and incorporated into the network geometry. The training error is initially zero by construction and remains so throughout the network topology transformation phase. The latter involves a small number of local topological transformations, such as splitting or merging of nodes and adding binary connections between them. The choice of operations to be carried out is only driven by optimization of expressivity at the local scale. What we are primarily looking for in a learning network is its ability to generalize, i.e. its capacity to correctly answer questions for which it has never learned the answers. We show on numerous examples of classification tasks that the networks generated by our algorithm systematically reach such a state of perfect generalization when the number of learned examples becomes sufficiently large. We report on the dynamics of the change of state and show that it is abrupt and has the distinctive characteristics of a first order phase transition, a phenomenon already observed for traditional learning networks and known as grokking. In addition to proposing a non-potential approach for the construction of learning networks, our algorithm makes it possible to rethink the grokking transition in a new light, under which acquisition of training data and topological structuring of data are completely decoupled phenomena.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。