用可训练的有理激活函数，让神经网络更高效、表达力更强。
Rational Neural Networks have Expressivity Advantages
- 提出可训练的低次有理激活函数，替代传统线性或光滑激活。
- 在相同精度下，有理网络参数量仅需对数双对数级，远少于传统网络。
- 实测性能媲美甚至超越经典激活函数，兼容主流训练流程。
我们研究了使用可训练低次有理激活函数的神经网络，发现其表达能力与参数效率均优于现代分段线性及光滑激活函数（如ELU、LeakyReLU、ReLU、SiLU、Mish等）。对于误差目标ε>0，我们建立逼近理论分离：任意由标准固定激活函数构成的网络，均可被有理激活网络以仅需poly(log log(1/ε))的规模增量实现一致逼近，而反向则在最坏情况下必然需要Ω(log(1/ε))参数。这一指数级差距在完整网络层面持续存在，并扩展至门控激活和Transformer式非线性结构。实践中，有理激活能无缝融入标准架构与训练流程，在相同模型与优化器下表现匹配或超越固定激活函数。
原文摘要 · Abstract (English)
We study neural networks with trainable low-degree rational activation functions and show that they are more expressive and parameter-efficient than modern piecewise-linear and smooth activations such as ELU, LeakyReLU, LogSigmoid, PReLU, ReLU, SELU, CELU, Sigmoid, SiLU, Mish, Softplus, Tanh, Softmin, Softmax, and LogSoftmax. For an error target of $\varepsilon>0$, we establish approximation-theoretic separations: Any network built from standard fixed activations can be uniformly approximated on compact domains by a rational-activation network with only $\mathrm{poly}(\log\log(1/\varepsilon))$ overhead in size, while the converse provably requires $Ω(\log(1/\varepsilon))$ parameters in the worst case. This exponential gap persists at the level of full networks and extends to gated activations and transformer-style nonlinearities. In practice, rational activations integrate seamlessly into standard architectures and training pipelines, allowing rationals to match or outperform fixed activations under identical architectures and optimizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。