通过模拟神经噪声,让人工网络自动形成模块化结构,提升泛化能力。
Modular connectivity in neural networks emerges from Poisson noise-motivated regularisation, and promotes robustness and compositional generalisation
- 用乘法形式的正则化项模拟神经活动噪声,促使网络自发模块化
- 模块化网络在噪声下更稳定,且能外推到训练数据之外
- 适合追求鲁棒性和组合泛化的模型设计者
大脑中的神经回路常表现为模块化架构,可分解复杂任务,实现组合泛化并减少灾难性遗忘。相比之下,人工神经网络(ANN)通常混合所有处理过程,因为模块化解在解空间中是趋于消失的子空间,难以被发现。本文受容错计算和真实神经元的泊松式放电启发,提出活动依赖的神经噪声结合非线性响应,能驱动网络涌现出符合模块化任务理解的解,即获得正确的世界模型。我们发现,这种噪声驱动的模块化可通过一种确定性正则化项再现,该正则化项以权重与激活的乘积形式组合,展现出线性网络或标准正则化无法捕捉的丰富现象。尽管模块化结构的出现需要足够多的训练样本(其数量随模块任务维度呈指数增长),但预模块化的ANN表现出显著更强的噪声鲁棒性,以及对训练数据外的场景进行良好泛化和外推的能力。本工作揭示了一种正则化方法与网络架构,可促进模块化涌现并带来功能性优势。
原文摘要 · Abstract (English)
Circuits in the brain commonly exhibit modular architectures that factorise complex tasks, resulting in the ability to compositionally generalise and reduce catastrophic forgetting. In contrast, artificial neural networks (ANNs) appear to mix all processing, because modular solutions are difficult to find as they are vanishing subspaces in the space of possible solutions. Here, we draw inspiration from fault-tolerant computation and the Poisson-like firing of real neurons to show that activity-dependent neural noise, combined with nonlinear neural responses, drives the emergence of solutions that reflect an accurate understanding of modular tasks, corresponding to acquisition of a correct world model. We find that noise-driven modularisation can be recapitulated by a deterministic regulariser that multiplicatively combines weights and activations, revealing rich phenomenology not captured in linear networks or by standard regularisation methods. Though the emergence of modular structure requires sufficiently many training samples (exponential in the number of modular task dimensions), we show that pre-modularised ANNs exhibit superior noise-robustness and the ability to generalise and extrapolate well beyond training data, compared to ANNs without such inductive biases. Together, our work demonstrates a regulariser and architectures that could encourage modularity emergence to yield functional benefits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。