用几何激活取代ReLU和归一化，实现100层网络稳定训练
Z-Plane Neural Networks: Bounded Geometric Activation Replaces ReLU and LayerNorm
- 将隐藏状态映射到超球面的相位束，用径向截断保持方向信息
- 100层MLP在MNIST上达98.34%准确率且无梯度消失
- 无需ReLU或层归一化，适合追求稳定性与可解释性的深度模型
现代深度神经网络依赖欧几里得标量激活（如ReLU）和全局归一化技术（如LayerNorm）来防止深层架构中的梯度不稳。然而，这些机制会导致死神经元、丢失关键方向信息并破坏特征表示的正交性。受生物轴突频率调制传输的启发，我们提出Z-Plane神经网络，将隐藏状态映射到超球面上的2D相位束。引入一种新的几何激活函数——径向截断（x / max(1, ||x||₂)），在限制能量幅度的同时保留相位（方向）。我们从数学上证明该各向同性激活函数具有1-Lipschitz连续性，并通过保持切向梯度防止梯度消失。实证表明，一个完全不含ReLU和LayerNorm的100层Z-Plane MLP在MNIST数据集上成功收敛，达到98.34%准确率且具备绝对数值稳定性，证明仅靠有界几何激活即可实现稳定深度学习。
原文摘要 · Abstract (English)
Modern deep neural networks rely on Euclidean scalar activations (e.g., ReLU) and global normalization techniques (e.g., LayerNorm) to prevent gradient instability in deep architectures. However, these mechanisms inherently cause dead neurons, discard critical directional information, and destroy the orthogonality of feature representations. Inspired by the frequency-modulation transmission of biological axons, we propose the Z-Plane Neural Network, which maps hidden states into 2D phasor bundles on a hypersphere. We introduce a novel geometric activation function, Radial Bounding($\mathbf{x} / \max(1, \|\mathbf{x}\|_2)$), which limits the energy magnitude while preserving the phase (direction). We demonstrate mathematically that this isotropic activation maintains 1-Lipschitz continuity and prevents gradient vanishing by preserving tangential gradients. Empirically, a 100-layer Z-Plane Multi-Layer Perceptron (MLP)-entirely devoid of ReLU and LayerNorm-successfully converges on the MNIST dataset with 98.34% accuracy and absolute numerical stability, proving that bounded geometric activation alone is sufficient for stable deep learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。