提出可学习的分段线性激活函数LSLU,提升模型泛化能力并加速训练。
Activation function optimization method: Learnable series linear units (LSLUs)
- 用可学习参数控制分段线性激活,动态适应每层训练阶段。
- 在CIFAR100上使VanillaNet准确率提升3.17%,且训练更快收敛。
- 适合追求高效高精度的深度学习模型优化者使用。
有效的激活函数引入非线性变换,增强神经网络的拟合能力,使其更适配真实数据分布。华为诺亚方舟实验室认为动态激活函数比静态函数更适合提升网络非线性能力,清华大学相关研究也支持动态调整激活函数。基于此,我们提出一种基于序列的可学习激活函数LSLU(Learnable Series Linear Units)。该方法引入可学习参数θ和ω,动态调节激活函数,适应当前层的训练阶段,提升模型泛化能力。核心思想是在每层激活中增加非线性,从而提升网络整体非线性表达能力。我们在CIFAR10、CIFAR100及特定任务数据集(如Silkworm)上评估了LSLU性能。实验分析了参数θ和ω的收敛行为及其对泛化的影响。结果表明,LSLU在多种任务中提升了原模型的泛化能力并加速训练。在VanillaNet训练中,参数θ先下降后上升趋于稳定,而ω呈现相反趋势。最终,LSLU在CIFAR100上使VanillaNet准确率提升3.17%(表3)。代码已公开于https://github.com/vontran2021/Learnable-series-linear-units-LSLU。
原文摘要 · Abstract (English)
Effective activation functions introduce non-linear transformations, providing neural networks with stronger fitting capa-bilities, which help them better adapt to real data distributions. Huawei Noah's Lab believes that dynamic activation functions are more suitable than static activation functions for enhancing the non-linear capabilities of neural networks. Tsinghua University's related research also suggests using dynamically adjusted activation functions. Building on the ideas of using fine-tuned activation functions from Tsinghua University and Huawei Noah's Lab, we propose a series-based learnable ac-tivation function called LSLU (Learnable Series Linear Units). This method simplifies deep learning networks while im-proving accuracy. This method introduces learnable parameters θ and ω to control the activation function, adapting it to the current layer's training stage and improving the model's generalization. The principle is to increase non-linearity in each activation layer, boosting the network's overall non-linearity. We evaluate LSLU's performance on CIFAR10, CIFAR100, and specific task datasets (e.g., Silkworm), validating its effectiveness. The convergence behavior of the learnable parameters θ and ω, as well as their effects on generalization, are analyzed. Our empirical results show that LSLU enhances the general-ization ability of the original model in various tasks while speeding up training. In VanillaNet training, parameter θ initially decreases, then increases before stabilizing, while ω shows an opposite trend. Ultimately, LSLU achieves a 3.17% accuracy improvement on CIFAR100 for VanillaNet (Table 3). Codes are available at https://github.com/vontran2021/Learnable-series-linear-units-LSLU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。