提出三类新激活函数，可稳定训练深层模型且支持高效微调。
Polynomial, trigonometric, and tropical activations
- 基于正交基的多项式、三角与热带激活函数，无需额外裁剪机制。
- 在GPT-2和ConvNeXt上实现与经典激活相当的性能，收敛稳定。
- 可通过插值逼近预训练模型激活，适合模型微调任务。
本文研究了基于正交基的激活函数族，包括埃尔米特多项式基、傅里叶三角函数基以及多项式基的热带化基。研究表明，通过简单的方差保持初始化，无需额外截断机制，这些激活函数可成功用于训练深度模型，如在OpenWebText上进行下一词预测的GPT-2，以及在ImageNet上进行图像分类的ConvNeXt。该方法有效缓解了多项式激活中常见的梯度爆炸与消失问题，为大规模学习任务的效率提升开辟了新路径。进一步分析表明，使用多项式激活的网络可被解释为多元多项式映射。利用埃尔米特插值，我们证明所提激活函数能同时匹配经典激活及其导数，从而在微调任务中表现优异。相关实现已开源：https://github.com/K-H-Ismail/torchortho。
原文摘要 · Abstract (English)
Which functions can be used as activations in deep neural networks? This article explores families of functions based on orthonormal bases, including the Hermite polynomial basis and the Fourier trigonometric basis, as well as a basis resulting from the tropicalization of a polynomial basis. Our study shows that, through simple variance-preserving initialization and without additional clamping mechanisms, these activations can successfully be used to train deep models, such as GPT-2 for next-token prediction on OpenWebText and ConvNeXt for image classification on ImageNet. Our work addresses the issue of exploding and vanishing activations and gradients, particularly prevalent with polynomial activations, and opens the door for improving the efficiency of large-scale learning tasks. Furthermore, our approach provides insight into the structure of neural networks, revealing that networks with polynomial activations can be interpreted as multivariate polynomial mappings. Finally, using Hermite interpolation, we show that our activations can closely approximate classical ones in pre-trained models by matching both the function and its derivative, making them especially useful for fine-tuning tasks. These activations are available in the torchortho library via: https://github.com/K-H-Ismail/torchortho.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。