RNN通过有限训练实现无限泛化，揭示了算法学习的内在机制。
Algorithm Development in Neural Networks: Insights from the Streaming Parity Task
- 在流式奇偶校验任务中，RNN通过代表态融合构建有限自动机。
- 有限训练后出现相变，实现对任意长序列的完美无限泛化。
- 适合研究神经网络算法生成与泛化能力的学者参考。
即使参数量极大，深度神经网络仍表现出惊人的泛化能力。现有研究多关注分布内平滑插值带来的泛化，但在某些场景下，神经网络还能外推至远超训练集范围的数据，甚至实现无限泛化，暗示其已学会解决任务的算法。本文以循环神经网络（RNN）在流式奇偶校验任务上的学习动态为案例，探讨算法发展的有效理论。该任务是定义于任意长度序列上的简单非线性任务。研究发现，经过充分的有限训练经验后，RNN会经历相变，实现完美的无限泛化。通过代表性动态的有效理论分析，我们发现了隐式的表示融合效应，可解释为构建了一个能复现任务的有限自动机。结果揭示了神经网络如何从有限训练经验中实现无限泛化的机制。
原文摘要 · Abstract (English)
Even when massively overparameterized, deep neural networks show a remarkable ability to generalize. Research on this phenomenon has focused on generalization within distribution, via smooth interpolation. Yet in some settings neural networks also learn to extrapolate to data far beyond the bounds of the original training set, sometimes even allowing for infinite generalization, implying that an algorithm capable of solving the task has been learned. Here we undertake a case study of the learning dynamics of recurrent neural networks (RNNs) trained on the streaming parity task in order to develop an effective theory of algorithm development. The streaming parity task is a simple but nonlinear task defined on sequences up to arbitrary length. We show that, with sufficient finite training experience, RNNs exhibit a phase transition to perfect infinite generalization. Using an effective theory for the representational dynamics, we find an implicit representational merger effect which can be interpreted as the construction of a finite automaton that reproduces the task. Overall, our results disclose one mechanism by which neural networks can generalize infinitely from finite training experience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。