用可学习的伯恩斯坦激活函数，实现指数级逼近速度与极低参数消耗。
Exponential Approximation Rates and Parameter Efficiency of Learnable Bernstein Activations
- 设计可学习的伯恩斯坦多项式激活，理论证明逼近误差随深度和阶数呈指数下降
- 在HIGGS和SUSY数据集上，参数减少超70%，训练次数少至ReLU的26%，损失降低45%
- 适合追求高精度与低资源消耗的模型设计，尤其适合科学计算场景
激活函数的选择从根本上影响深度神经网络的表达能力与参数效率，但现有常用激活函数缺乏相关性质的严格理论保证。本文对采用可学习伯恩斯坦多项式激活的DeepBern-Nets（DBNs）进行了理论分析，证明其逼近误差随网络深度$L$和多项式阶数$n$以$/mathcal{O}(n^{-L})$的速率衰减，远快于ReLU架构的多项式速率，同时保持完全可微。通过在大型科学数据集（HIGGS和SUSY）上开展1,344次实验，对比了DBNs与ReLU、Leaky ReLU、SELU和GeLU。结果表明，DBNs在多数架构中实现超过70%的参数压缩——在大规模下可达99.9%，收敛至ReLU最终损失仅需其26%的训练轮次，并实现最高达45%的更低最终损失。这些优势在所有测试激活函数中均成立，证实其性能提升源于可学习多项式结构，而非单纯的平滑性。
原文摘要 · Abstract (English)
The choice of activation function fundamentally shapes the representational capacity and parameter efficiency of deep neural networks, yet most widely used activations lack rigorous theoretical guarantees on these properties. We provide a theoretical analysis of DeepBern-Nets (DBNs) -- networks employing learnable Bernstein polynomial activations -- showing that their approximation error decays with the network depth $L$ and the polynomial order $n$ with a rate of $\mathcal{O}(n^{-L})$, exponentially faster than the polynomial rate of ReLU architectures while remaining fully differentiable. We validate these predictions through $1{,}344$ experiments on large scientific datasets (HIGGS and SUSY), comparing DBNs against ReLU, Leaky ReLU, SELU, and GeLU. DBNs achieve over $70\%$ parameter reduction across the majority of architectures -- reaching $99.9\%$ at scale -- converge to ReLU's final loss in as few as $26\%$ of the training epochs, and attain up to $45\%$ lower final loss. These advantages hold over all tested activations, confirming that DBN's gains stem from the learnable polynomial structure rather than mere smoothness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。