用特定激活函数，小网络也能轻松学会生命游戏规则
It's Much Easier for Neural Networks to learn Game of Life Dynamics with the Right Activation Function: Polynomial Kolmogorov-Arnold Networks
- 采用二次多项式激活函数，提升网络对生命游戏的建模能力
- 2阶多项式网络在无权重学习时仍能稳定复现生命游戏动态
- 为科学机器学习提供可解释的简化测试场景
先前研究发现，能可靠学习康威生命游戏的神经网络规模，远大于理论上最小可表示该元胞自动机的网络。将神经网络学习视为搜索过程，暗示需要足够大的网络以包含幸运初始化的子网络（即‘赢票’）。本文重新将生命游戏规则学习视为学习问题，认为合适的归纳偏置可使小规模网络更易实现。我们发现，多种替代激活函数显著优于默认的ReLU，尤其是二阶多项式激活函数，可在有无权重学习条件下一致复现生命游戏动态。结果表明，匹配任务特性的学习策略至关重要，挑战了盲目扩大模型规模的默认做法。我们倡导将元胞自动机作为发展机器学习于科学、物理驱动深度学习及可解释学习策略的简易测试域。
原文摘要 · Abstract (English)
Previous work has found a gap between the scale of neural networks that reliably learn Conway's Game of Life, and minimal networks capable of representing the classic cellular automaton with hard-coded parameter values. Viewing neural network learning as a search process suggests a dependence on networks large enough to contain sub-networks with lucky initializations (sometimes known as 'winning tickets') that actually learn the task. In this work, we reorient our perspective from discovering Life rules as a search problem back to a learning problem, and reason that with fitting inductive biases, the problem should be much more amenable to minimal networks. We find that network variants with several alternative activation functions meaningfully outperform the default choice of Rectified Linear Units, and in particular, that a 2nd degree polynomial activation function consistently learns Life dynamics with or without the benefit of learning neural weights. Our results provide an informative demonstration of the benefits of matching learning to the task at hand and challenge the easy default choice of scale for all problems. In particular, we advocate for the use of cellular automata as simple test domains for developing strategies that can benefit machine learning for science, physics-based deep learning, and interpretable machine learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。