用局部规则自动生成不同架构的神经网络权重,无需反向传播。
Architecture Generalization with MetaNCA

- 通过局部交互的元规则网络,迭代更新任务网络权重。
- 可生成200万参数的MLP/CNN/ResNet,在MNIST和CIFAR-100上有效。
- 训练时引入架构多样性,提升对未见结构的泛化能力。
自组织是生命体通过个体组件在局部信息驱动下协同作用而产生的特性。生物神经元通过突触传递的局部交互,能高效学习并随生命期调整连接。受此启发,神经细胞自动机(NCA)模型仅通过局部更新规则即可实现形态发生的学习,展现出多轮更新下的稳定性与抗扰性。本文提出元神经细胞自动机(MetaNCA),一种学习局部规则以自组织人工神经网络权重的框架。该规则网络通过计算图上的局部交互,迭代更新任务网络权重。我们设计了新的权重变换器(Weight Transformer)架构,采用线性注意力聚合邻近权重与隐藏状态信号。训练完成后,规则网络可生成无需反向传播的多样化网络架构。实验表明,MetaNCA可在MNIST和CIFAR-100上生成前馈MLP、CNN和ResNet,规模达200万参数,并在未见过的架构上实现良好泛化;训练阶段引入架构多样性可进一步增强泛化能力。
原文摘要 · Abstract (English)
Self-organization is an emergent property of life, driven by the collective behavior of individual components acting on local information. Biological neurons, through local interactions transmitted through synapses, are able to learn efficiently and can adapt their connections over an organism's lifespan. Motivated by these desirable properties of adaptability and local interaction, neural cellular automata (NCA) models have been successful at learning morphogenesis solely through local update rules, demonstrating stability over many updates and robustness to perturbations. In this work, we introduce Meta Neural Cellular Automata (MetaNCA), a framework that learns local rules which self-organize the weights of artificial neural networks. A learned rule network iteratively updates the weights of a task network using only local interactions on the computation graph. We propose a novel Weight Transformer architecture for the local rule network, which uses linear attention to aggregate signals from neighboring weights and hidden states. Once trained, the rule network generates task networks of diverse architectures without backpropagation. We show that MetaNCA generates weights for feedforward MLPs, CNNs, and ResNets on MNIST and CIFAR-100, scaling to networks of 2 million parameters. We further show that MetaNCA generalizes to architectures not seen during meta-training, and that architectural diversity in the training phase strengthens this generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。