用高斯混合模型设计可训练非线性模块，提升多种神经网络性能
Rethinking Nonlinearity: Trainable Gaussian Mixture Modules for Modern Neural Architectures
- 基于高斯混合模型设计可微分的非线性模块，灵活参数化投影
- 在MLP、CNN、注意力机制等架构中均显著优于标准基线
- 适合需要增强非线性表达能力的现代神经网络应用
从MLP、CNN到基于注意力的Transformer，现有神经网络普遍由线性组合后接非线性激活函数（如ReLU、Sigmoid）构成。尽管有效，但其非线性能力受限于激活函数选择。本文提出高斯混合启发的非线性模块（GMNM），借鉴高斯混合模型（GMM）的密度逼近能力和高斯核的距离特性。通过放松概率约束并采用灵活的高斯投影参数化，GMNM可无缝嵌入各类神经架构，并支持端到端梯度训练。实验表明，在MLP、CNN、注意力机制及LSTM中引入GMNM均能持续提升性能，验证其在提升模型效率与精度方面的强大潜力。
原文摘要 · Abstract (English)
Neural networks in general, from MLPs and CNNs to attention-based Transformers, are constructed from layers of linear combinations followed by nonlinear operations such as ReLU, Sigmoid, or Softmax. Despite their strength, these conventional designs are often limited in introducing non-linearity by the choice of activation functions. In this work, we introduce Gaussian Mixture-Inspired Nonlinear Modules (GMNM), a new class of differentiable modules that draw on the universal density approximation Gaussian mixture models (GMMs) and distance properties (metric space) of Gaussian kernal. By relaxing probabilistic constraints and adopting a flexible parameterization of Gaussian projections, GMNM can be seamlessly integrated into diverse neural architectures and trained end-to-end with gradient-based methods. Our experiments demonstrate that incorporating GMNM into architectures such as MLPs, CNNs, attention mechanisms, and LSTMs consistently improves performance over standard baselines. These results highlight GMNM's potential as a powerful and flexible module for enhancing efficiency and accuracy across a wide range of machine learning applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。