让神经网络自动挑选最优激活函数，提升模型适应性。
FlexAct: Why Learn when you can Pick?
- 用Gumbel-Softmax实现可微的离散激活函数选择
- 在合成数据上始终选对最适合的激活函数
- 适合需要灵活架构的复杂任务场景
学习激活函数已成为深度学习中一个有前景的方向，使网络能够根据任务需求自适应调整激活机制。本文提出一种新框架，利用Gumbel-Softmax技巧，在训练过程中对预定义的一组激活函数进行离散但可微的选择。该方法独立于输入动态学习最优激活函数，从而提升预测精度与结构灵活性。在合成数据集上的实验表明，模型能持续选择最合适的激活函数，验证了其有效性。该工作将理论进展与实际应用结合，为复杂学习场景中的更自适应、模块化神经架构铺平道路。
原文摘要 · Abstract (English)
Learning activation functions has emerged as a promising direction in deep learning, allowing networks to adapt activation mechanisms to task-specific demands. In this work, we introduce a novel framework that employs the Gumbel-Softmax trick to enable discrete yet differentiable selection among a predefined set of activation functions during training. Our method dynamically learns the optimal activation function independently of the input, thereby enhancing both predictive accuracy and architectural flexibility. Experiments on synthetic datasets show that our model consistently selects the most suitable activation function, underscoring its effectiveness. These results connect theoretical advances with practical utility, paving the way for more adaptive and modular neural architectures in complex learning scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。