剪枝能自发产生神经网络中的信息流向有序性。
Pruning Increases Orderedness in Recurrent Computation
- 用剪枝技术在全连接层中诱导出类似前馈的有序信息流。
- 不同随机种子下剪枝均提升信息流拓扑有序度且性能不变。
- 适合关注神经网络结构演化与生物启发设计的研究者。
受生物大脑中广泛存在的循环回路启发，我们研究了方向性作为人工神经网络有益先验假设的程度。将方向性定义为神经元间拓扑有序的信息流动，我们形式化了一个全连接感知机层（数学上等价于权值共享的循环神经网络），并证明方向性——现代前馈网络的特征——可通过适当的剪枝技术实现，而非硬编码。在不同随机种子下，我们的剪枝方案成功在不损害性能的前提下提升了神经元间信息流的拓扑有序性，表明方向性并非学习的必要条件，而是可通过梯度下降与稀疏化发现的有利先验。
原文摘要 · Abstract (English)
Inspired by the prevalence of recurrent circuits in biological brains, we investigate the degree to which directionality is a helpful inductive bias for artificial neural networks. Taking directionality as topologically-ordered information flow between neurons, we formalise a perceptron layer with all-to-all connections (mathematically equivalent to a weight-tied recurrent neural network) and demonstrate that directionality, a hallmark of modern feed-forward networks, can be induced rather than hard-wired by applying appropriate pruning techniques. Across different random seeds our pruning schemes successfully induce greater topological ordering in information flow between neurons without compromising performance, suggesting that directionality is not a prerequisite for learning, but may be an advantageous inductive bias discoverable by gradient descent and sparsification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。