用周期性激活函数让小模型高效解决复杂任务
From Taylor Series to Fourier Synthesis: The Periodic Linear Unit
- 提出基于正弦波的PLU激活函数，具备非单调周期特性
- 仅两个PLU神经元即可准确分类螺旋数据，传统网络无法做到
- 适合追求高参数效率的模型设计者，推动神经网络范式革新
当前主流神经网络依赖ReLU等单调递增激活函数，虽有效但需大规模参数来逼近复杂函数。本文提出周期线性单元（PLU），一种基于可学习正弦波的周期性非单调激活函数，通过其构造和名为“排斥重参数化”的创新机制，确保数值稳定并防止退化为线性函数。实验表明，仅含两个PLU神经元的极简MLP即可解决螺旋分类任务，而使用标准激活的同等结构则无法实现。这标志着从分段泰勒逼近到强大傅里叶函数合成的范式转变，将智能嵌入神经元本身，实现指数级的参数效率提升。
原文摘要 · Abstract (English)
The dominant paradigm in modern neural networks relies on simple, monotonically-increasing activation functions like ReLU. While effective, this paradigm necessitates large, massively-parameterized models to approximate complex functions. In this paper, we introduce the Periodic Linear Unit (PLU), a learnable sine-wave based activation with periodic non-monotonicity. PLU is designed for maximum expressive power and numerical stability, achieved through its formulation and a paired innovation we term Repulsive Reparameterization, which prevents the activation from collapsing into a non-expressive linear function. We demonstrate that a minimal MLP with only two PLU neurons can solve the spiral classification task, a feat impossible for equivalent networks using standard activations. This suggests a paradigm shift from networks as piecewise Taylor-like approximators to powerful Fourier-like function synthesizers, achieving exponential gains in parameter efficiency by placing intelligence in the neuron itself.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。