并行层归一化让神经网络具备通用逼近能力，突破传统归一化限制。
Parallel Layer Normalization for Universal Approximation
- 在两层线性层间加入并行层归一化，构建可逼近任意函数的网络结构
- 理论证明并行层归一化网络在无穷范数和Sobolev范数下具有最优逼近率
- 适用于Transformer等主流架构，对模型设计有重要启发意义
本文研究了结合层归一化（LN）与线性层的神经网络的逼近能力。证明仅含标准层归一化的网络表达力受限，而使用两个线性层间插入并行层归一化（PLN）的结构（称作PLN-Nets）可实现通用逼近。进一步分析了浅层与深层PLN-Nets在$L^\infty$范数及Sobolev范数下的逼近速率。分析结果扩展至RMSNorm，并覆盖标准MLP与位置前馈网络（Transformer核心模块）。最后通过实验探索了PLN-Nets的其他潜力。
原文摘要 · Abstract (English)
This paper studies the approximation capabilities of neural networks that combine layer normalization (LN) with linear layers. We prove that networks consisting of two linear layers with parallel layer normalizations (PLNs) inserted between them (referred to as PLN-Nets) achieve universal approximation, whereas architectures that use only standard LN exhibit strictly limited expressive power.We further analyze approximation rates of shallow and deep PLN-Nets under the $L^\infty$ norm as well as in Sobolev norms. Our analysis extends beyond LN to RMSNorm, and from standard MLPs to position-wise feed-forward networks, the core building blocks used in RNNs and Transformers.Finally, we provide empirical experiments to explore other possible potentials of PLN-Nets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。