用伯恩斯坦多项式设计平滑激活函数,兼顾稳定训练与高效推理。
Universal Smoothness via Bernstein Polynomials: A Constructive Approximation Approach for Activation Functions
- 基于构造性逼近理论,用伯恩斯坦多项式实现平滑过渡区。
- 理论保证梯度连续且不扩张,避免深层网络梯度爆炸。
- 在视觉模型上性能超前沿方法,计算内存开销更低。
深度神经网络的性能高度依赖非线性激活函数的设计,但现有方法难以兼顾优化稳定性与计算效率。分段线性函数虽推理快,却因原点不可导导致优化不稳定;光滑函数通常依赖超越运算,带来显著计算开销。本文提出基于构造性逼近理论的通用平滑框架,引入伯恩斯坦线性单元(BerLU)。该激活函数利用伯恩斯坦多项式构建可导的二次过渡区,有效消除奇点,同时保持分段线性结构。理论分析表明,该方法保证严格连续可导,且Lipschitz常数不超过1,确保梯度传播稳定,防止深层网络中的梯度爆炸。在代表性视觉变换器和卷积神经网络架构上的全面实验验证了该方法在标准图像分类基准上持续优于现有最优基线,同时实现更优的计算与内存效率。
原文摘要 · Abstract (English)
The efficacy of deep neural networks is heavily reliant on the design of non-linear activation functions, yet existing approaches often struggle to balance optimization stability with computational efficiency. While piecewise linear functions offer inference speed, they suffer from optimization instability due to non-differentiability at the origin, whereas smooth counterparts typically incur significant computational overhead through their reliance on transcendental operations. To address these limitations, this paper proposes a general smoothing framework based on constructive approximation theory and introduces the Bernstein Linear Unit (BerLU). This novel activation function utilizes Bernstein polynomials to construct a differentiable quadratic transition region that effectively eliminates singularities while maintaining a piecewise linear structure. Theoretical analysis demonstrates that the proposed method guarantees strictly continuous differentiability and a non-expansive Lipschitz constant of one, which ensures stable gradient propagation and prevents the gradient explosion problems common in deep architectures. Comprehensive empirical evaluations across representative Vision Transformer and Convolutional Neural Network architectures confirm that this approach consistently outperforms state-of-the-art baselines on standard image classification benchmarks while delivering superior computational and memory efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。