提出新型非线性神经元Paon,用更少层数实现更强表征能力。
Padé Neurons for Efficient Neural Models
- 基于帕德逼近设计可学习非线性函数的神经元,每层自适应生成不同非线性映射。
- 在图像超分、压缩和分类任务中,用更少层数达到或超过经典ResNet性能。
- 兼容已有神经元模型,适合追求高效轻量网络的研究者与开发者。
神经网络普遍采用麦卡洛克-皮茨神经元模型,即线性变换后接逐点非线性激活函数。已有研究提出了二次神经元、广义操作神经元、生成神经元和超神经元等内在非线性模型,其非线性能力优于传统逐点激活函数。本文提出一种受帕德逼近启发的新颖非线性神经元模型——帕德神经元(Paons),具有非线性多样性:每个Paon可学习输入的不同非线性函数;且层效率高:以更少层数实现更强非线性,优于分段线性近似。此外,所有先前提出的神经元模型均为Paons的特例,因此任意网络中的神经元均可被替换为Paons。值得注意的是,已有研究将帕德逼近用于广义逐点激活函数,与本模型本质不同。实验中,我们在基于ResNet架构的多个经典图像超分辨率、压缩和分类模型中用Paons替代传统神经元。全面实验结果表明,使用Paons构建的模型在层数更少的情况下性能不低于甚至优于原模型。PyTorch代码已在GitHub开源:https://github.com/onur-keles/Paon。
原文摘要 · Abstract (English)
Neural networks commonly employ the McCulloch-Pitts neuron model, which is a linear model followed by a point-wise non-linear activation. Various researchers have already advanced inherently non-linear neuron models, such as quadratic neurons, generalized operational neurons, generative neurons, and super neurons, which offer stronger non-linearity compared to point-wise activation functions. In this paper, we introduce a novel and better non-linear neuron model called Padé neurons (Paons), inspired by Padé approximants. Paons offer several advantages, such as diversity of non-linearity, since each Paon learns a different non-linear function of its inputs, and layer efficiency, since Paons provide stronger non-linearity in much fewer layers compared to piecewise linear approximation. Furthermore, Paons include all previously proposed neuron models as special cases, thus any neuron model in any network can be replaced by Paons. We note that there has been a proposal to employ the Padé approximation as a generalized point-wise activation function, which is fundamentally different from our model. To validate the efficacy of Paons, in our experiments, we replace classic neurons in some well-known neural image super-resolution, compression, and classification models based on the ResNet architecture with Paons. Our comprehensive experimental results and analyses demonstrate that neural models built by Paons provide better or equal performance than their classic counterparts with a smaller number of layers. The PyTorch implementation code for Paon is open-sourced at https://github.com/onur-keles/Paon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。