让神经网络的激活函数自动优化，提升模型精度与效率
Developing Training Procedures for Piecewise-linear Spline Activation Functions in Neural Networks
- 用可学习的分段线性样条函数替代传统激活函数
- 在前馈网络中误差降低94%，卷积网络降低51%
- 适合追求模型效率与精度的研究者参考
神经网络中的激活函数通常采用经验验证的静态函数，如ReLU、tanh或sigmoid。通过优化网络中激活函数的形状，可训练出参数更高效、准确率更高的模型。本文提出并对比了9种训练方法，探索使用参数化线性B样条激活函数时的双重优化动态。实验表明，在前馈网络中模型误差最高降低94%，在卷积网络中降低51%，相比传统的ReLU模型。这些性能提升以增加开发复杂度和模型推理延迟为代价。
原文摘要 · Abstract (English)
Activation functions in neural networks are typically selected from a set of empirically validated, commonly used static functions such as ReLU, tanh, or sigmoid. However, by optimizing the shapes of a network's activation functions, we can train models that are more parameter-efficient and accurate by assigning more optimal activations to the neurons. In this paper, I present and compare 9 training methodologies to explore dual-optimization dynamics in neural networks with parameterized linear B-spline activation functions. The experiments realize up to 94% lower end model error rates in FNNs and 51% lower rates in CNNs compared to traditional ReLU-based models. These gains come at the cost of additional development and training complexity as well as end model latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。