发现有限宽度网络中信息传播的分形边界，揭示深层网络动态本质复杂性。
Revisiting Deep Information Propagation: Fractal Frontier and Finite-size Effects
- 通过分析有限宽度网络，发现信息传播边界呈分形结构。
- 在多层感知机与卷积网络中均观察到相同分形行为。
- 揭示深度与分离性/鲁棒性权衡的关键作用，适合研究网络动态者参考。
信息传播描述了输入相关性在深度神经网络各层间的演化过程。该框架通常基于均值场理论研究，假设网络无限宽。然而，这一假设在实际有限尺寸网络中不成立。本文研究随机初始化的有限宽度神经网络中的信息传播，发现有序与混沌区域之间的边界呈现分形结构，揭示了神经网络动态的根本复杂性，且该现象独立于输入数据和优化过程。为将分析拓展至多层感知机之外，我们采用最近提出的基于傅里叶的结构化变换，证明卷积神经网络的信息传播也遵循相同规律。实践中，本研究强调了网络深度在分离性与鲁棒性权衡中的关键作用。
原文摘要 · Abstract (English)
Information propagation characterizes how input correlations evolve across layers in deep neural networks. This framework has been well studied using mean-field theory, which assumes infinitely wide networks. However, these assumptions break down for practical, finite-size networks. In this work, we study information propagation in randomly initialized neural networks with finite width and reveal that the boundary between ordered and chaotic regimes exhibits a fractal structure. This shows the fundamental complexity of neural network dynamics, in a setting that is independent of input data and optimization. To extend this analysis beyond multilayer perceptrons, we leverage recently introduced Fourier-based structured transforms, and show that information propagation in convolutional neural networks also follow the same behavior. In practice, our investigation highlights the importance of finite network depth with respect to the tradeoff between separation and robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。