用进化算法优化图像分类用激活函数,效果优于现有方法。
Activation Function Optimization Scheme for Image Classification
- 通过进化算法自动搜索适合图像分类的激活函数。
- 在28个测试场景中92.8%情况下表现优于现有标准函数。
- 最优函数为-x·erf(e⁻ˣ),适用于各类图像模型与数据集。
激活函数对深度神经网络的动态性、收敛性和性能有显著影响。现有高性能激活函数多依赖人工设计,仅Swish采用强化学习搜索。本文提出一种专用于图像分类任务的进化优化方法,旨在发现超越当前最优的激活函数。通过该框架,获得一系列高性能函数,统称为指数误差线性单元(EELU)。所提函数在五种主流网络架构(ResNet50、AlexNet、VGG16、MobileNet、Compact Convolutional Transformer)及八个标准数据集(CIFAR10、Imagenette、MNIST、Fashion MNIST、Beans、Colorectal Histology、CottonWeedID15、TinyImageNet)上评估,涵盖从通用视觉到农业与医学图像应用。通过弗里德曼检验,结果表明该优化方案生成的函数在28组对比中92.8%情况下优于现有标准函数,其中最优函数为 -x·erf(e⁻ˣ)。
原文摘要 · Abstract (English)
Activation function has a significant impact on the dynamics, convergence, and performance of deep neural networks. The search for a consistent and high-performing activation function has always been a pursuit during deep learning model development. Existing state-of-the-art activation functions are manually designed with human expertise except for Swish. Swish was developed using a reinforcement learning-based search strategy. In this study, we propose an evolutionary approach for optimizing activation functions specifically for image classification tasks, aiming to discover functions that outperform current state-of-the-art options. Through this optimization framework, we obtain a series of high-performing activation functions denoted as Exponential Error Linear Unit (EELU). The developed activation functions are evaluated for image classification tasks from two perspectives: (1) five state-of-the-art neural network architectures, such as ResNet50, AlexNet, VGG16, MobileNet, and Compact Convolutional Transformer which cover computationally heavy to light neural networks, and (2) eight standard datasets, including CIFAR10, Imagenette, MNIST, Fashion MNIST, Beans, Colorectal Histology, CottonWeedID15, and TinyImageNet which cover from typical machine vision benchmark, agricultural image applications to medical image applications. Finally, we statistically investigate the generalization of the resultant activation functions developed through the optimization scheme. With a Friedman test, we conclude that the optimization scheme is able to generate activation functions that outperform the existing standard ones in 92.8% cases among 28 different cases studied, and $-x\cdot erf(e^{-x})$ is found to be the best activation function for image classification generated by the optimization scheme.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。