让神经网络每个神经元自选激活函数,提升性能且不增加推理负担。
SmartMixed: A Two-Phase Training Strategy for Adaptive Activation Function Learning in Neural Networks
- 分两阶段训练:先动态选择,再固定最优激活函数。
- 在MNIST上表现优于单一固定激活函数的模型。
- 揭示不同层神经元对激活函数有不同偏好,适合想优化网络结构的研究者。
激活函数的选择对神经网络至关重要,但多数架构仍对所有神经元使用固定的统一激活函数。本文提出SmartMixed,一种新型两阶段训练策略,使网络能够学习每个神经元的最优激活函数,同时保持推理时的计算效率。第一阶段,神经元通过可微硬混合机制从一组候选函数(ReLU、Sigmoid、Tanh、Leaky_ReLU、ELU、SELU)中自适应选择;第二阶段,根据学习到的选择结果固定各神经元的激活函数,形成支持后续训练且具备优化向量化操作的高效网络。我们在不同架构的前馈神经网络上对MNIST数据集进行了评估。分析显示,不同层的神经元对激活函数存在显著偏好,揭示了神经网络内部的功能多样性。实验表明,SmartMixed通过允许神经元自主选择更优激活函数,有效提升了网络训练效果,超越了使用单一先进激活函数的基准模型。
原文摘要 · Abstract (English)
The choice of activation function plays a critical role in neural networks, yet most architectures still rely on fixed, uniform activation functions across all neurons. We introduce SmartMixed, a novel two-phase training strategy that allows networks to learn optimal per-neuron activation functions while preserving computational efficiency at inference. In the first phase, neurons adaptively select from a pool of candidate activation functions (ReLU, Sigmoid, Tanh, Leaky_ReLU, ELU, SELU) using a differentiable hard mixture mechanism. In the second phase, each neuron's activation function is fixed according to the learned selection, resulting in a computationally efficient network that supports continued training with optimized vectorized operations. We evaluate SmartMixed on the MNIST dataset using feedforward neural networks of different architectures. Our analysis reveals that neurons in different layers exhibit distinct preferences for activation functions, providing insights into the functional diversity within neural architectures. We also demonstrated that SmartMixed effectively trains the network by allowing neurons to select their preferred activation functions, competing against models using a single fixed state-of-the-art activation function.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。