通过积分设计可训练激活函数，提升大模型性能。
Deriving Activation Functions Using Integration
- 从梯度出发，用积分推导出可训练的分段激活函数。
- 在1.1B和3B参数模型上，困惑度低于ReLU²和SwiGLU。
- 适合追求高性能的大模型研究者使用。
本文提出一种新方法，通过关注激活函数的梯度并利用积分推导其形式。我们引入了可训练的扩展指数线性单元（xIELU），该函数通过对指数线性单元（ELU）施加可训练仿射变换后积分得到。xIELU具备两个关键梯度特性：(1) 正输入时具有可训练且线性递增的梯度，类似平方ReLU（ReLU²）；(2) 负输入时梯度可取负值且可训练，受扩展SiLU（xSiLU）启发。概念上，xIELU可视为将ReLU²拓展至处理负输入。其可训练参数能自适应降低深层网络中高级表示的非线性。在1.1B与3B参数的Llama模型上，基于125B条细粒度网页数据训练，xIELU在相同计算成本和参数量下，相比ReLU²和SwiGLU实现更低的困惑度。参考实现已公开于https://github.com/Anonymous5823/xielu。
原文摘要 · Abstract (English)
Our work proposes a novel approach to designing activation functions by focusing on their gradients and deriving the corresponding activation functions using integration. We introduce the Expanded Integral of the Exponential Linear Unit (xIELU), a trainable piecewise activation function derived by integrating trainable affine transformations applied to the Exponential Linear Unit (ELU). xIELU combines two key properties for the gradient: (1) a trainable and linearly increasing gradient for positive inputs, similar to Squared ReLU (ReLU$^2$), and (2) a trainable gradient that can take negative values for negative inputs, inspired by Expanded SiLU (xSiLU). Conceptually, xIELU can be viewed as an extension of ReLU$^2$ to handle negative inputs. The trainable parameters in xIELU allow it to adaptively reduce its nonlinearity for higher-level representations deeper in the network. In experiments with 1.1B and 3B parameter Llama models trained on 125B tokens of FineWeb Edu, xIELU achieves lower perplexity compared to popular activation functions like ReLU$^2$ and SwiGLU when matched for the same compute cost and parameter count. A reference implementation is available at https://github.com/Anonymous5823/xielu.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。