激活函数曲率影响对抗鲁棒性，适度曲率最有利
Why the Maximum Second Derivative of Activations Matters for Adversarial Robustness
- 用可调曲率的激活函数家族，系统研究曲率对模型的影响
- 曲率在4到10之间时，对抗鲁棒性最佳，呈非单调关系
- 曲率过低或过高都会损害鲁棒性，适合关注模型稳定性的研究者
本文研究激活函数曲率（以最大二阶导数 $\ ext{max}|σ''|$ 表示）在对抗鲁棒性中的关键作用。通过递归可调曲率激活族（RCT-AF），可精确控制参数 $α$ 与 $β$ 来调节曲率，系统分析其影响。研究发现存在根本性权衡：曲率不足限制模型表达力，而过度曲率会放大损失函数的归一化海森对角线范数，导致尖锐极小值，阻碍鲁棒泛化。因此，对抗鲁棒性呈现非单调变化，最优鲁棒性始终出现在 $\ ext{max}|σ''|$ 介于4至10区间，该结论在多种网络结构、数据集及对抗训练方法中均成立。我们提供了关于激活曲率如何影响损失函数海森对角元素的理论见解，并实验验证了归一化海森对角线范数随 $\ ext{max}|σ''|$ 呈现U形变化，在最优鲁棒性区间取得最小值，从而支持所提机制。
原文摘要 · Abstract (English)
This work investigates the critical role of activation function curvature -- quantified by the maximum second derivative $\max|σ''|$ -- in adversarial robustness. Using the Recursive Curvature-Tunable Activation Family (RCT-AF), which enables precise control over curvature through parameters $α$ and $β$, we systematically analyze this relationship. Our study reveals a fundamental trade-off: insufficient curvature limits model expressivity, while excessive curvature amplifies the normalized Hessian diagonal norm of the loss, leading to sharper minima that hinder robust generalization. This results in a non-monotonic relationship where optimal adversarial robustness consistently occurs when $\max|σ''|$ falls within 4 to 10, a finding that holds across diverse network architectures, datasets, and adversarial training methods. We provide theoretical insights into how activation curvature affects the diagonal elements of the hessian matrix of the loss, and experimentally demonstrate that the normalized Hessian diagonal norm exhibits a U-shaped dependence on $\max|σ''|$, with its minimum within the optimal robustness range, thereby validating the proposed mechanism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。