提出新激活函数，提升模型抗攻击能力与泛化性能。
RCR-AF: Enhancing Model Generalization via Rademacher Complexity Reduction Activation Function
- 基于径向复杂度理论设计新型激活函数，可调控模型稀疏性与容量。
- 在标准训练和对抗训练中均优于ReLU、GELU等主流激活函数。
- 适合关注模型鲁棒性与泛化能力的深度学习研究者使用。
尽管深度神经网络广泛应用，但在安全敏感场景中仍极易受到对抗攻击。本文探究激活函数作为提升模型鲁棒性的关键但未充分研究的组件。提出一种径向复杂度降低激活函数（RCR-AF），该函数结合了GELU的平滑性、梯度稳定性及负信息保留特性，同时具备ReLU的单调性，并通过两个超参数 $α$ 和 $γ$ 控制模型稀疏性和容量。理论分析表明，这两个参数可直接调节模型的径向复杂度，为增强鲁棒性提供理论依据。大量实验证明，无论在标准训练下的干净准确率，还是对抗训练中的抗攻击能力，RCR-AF 均持续优于ReLU、GELU和Swish等常见激活函数。
原文摘要 · Abstract (English)
Despite their widespread success, deep neural networks remain critically vulnerable to adversarial attacks, posing significant risks in safety-sensitive applications. This paper investigates activation functions as a crucial yet underexplored component for enhancing model robustness. We propose a Rademacher Complexity Reduction Activation Function (RCR-AF), a novel activation function designed to improve both generalization and adversarial resilience. RCR-AF uniquely combines the advantages of GELU (including smoothness, gradient stability, and negative information retention) with ReLU's desirable monotonicity, while simultaneously controlling both model sparsity and capacity through built-in clipping mechanisms governed by two hyperparameters, $α$ and $γ$. Our theoretical analysis, grounded in Rademacher complexity, demonstrates that these parameters directly modulate the model's Rademacher complexity, offering a principled approach to enhance robustness. Comprehensive empirical evaluations show that RCR-AF consistently outperforms widely-used alternatives (ReLU, GELU, and Swish) in both clean accuracy under standard training and in adversarial robustness within adversarial training paradigms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。