用随机激活函数提升大模型推理速度与生成多样性
Stochastic activations
- 前向传播时随机选SILU或RELU，解决RELU梯度停滞问题
- 预训练用随机激活，推理用RELU，FLOPs降低实现加速
- 生成文本多样性更高，适合追求创意输出的场景
我们提出随机激活策略，在大语言模型的前馈层中基于伯努利采样随机选择SILU或RELU非线性函数。该方法规避了RELU在负输入下梯度无法流动的优化难题。一方面，预训练阶段使用随机激活，微调后推理时采用RELU，可生成稀疏潜在表示，显著降低推理时的计算量（FLOPs），在CPU和GPU上均实现明显加速，效果优于从头训练的RELU模型。另一方面，在序列生成任务中，该策略表现良好：生成文本多样性更高，仅略逊于最优确定性非线性函数SILU结合温度采样的表现，为提升生成多样性提供新途径。
原文摘要 · Abstract (English)
We introduce stochastic activations. This novel strategy randomly selects between several non-linear functions in the feed-forward layer of a large language model. In particular, we choose between SILU or RELU depending on a Bernoulli draw. This strategy circumvents the optimization problem associated with RELU, namely, the constant shape for negative inputs that prevents the gradient flow. We leverage this strategy in two ways: (1) We use stochastic activations during pre-training and fine-tune the model with RELU, which is used at inference time to provide sparse latent vectors. This reduces the inference FLOPs and translates into a significant speedup on CPU and GPU. This leads to better results than training from scratch with the RELU activation function. (2) We evaluate stochastic activations for sequence generation. This strategy performs reasonably well: it has higher diversity and has only slightly inferior performance to the best deterministic non-linearity, SILU, combined with temperature sampling. This provides an alternative way to increase the diversity of generated text.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。