揭示对抗训练中正则化范数选择的几何规律,指导高维数据下模型鲁棒性设计。
On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Bounds
- 通过渐近分析推导不同对抗攻击与正则化范数下的鲁棒解结构。
- 给出统一收敛界,量化了扰动大小与最优正则化范数的关系。
- 适用于数据稀疏时的对抗训练优化,尤其适合研究鲁棒性机制的研究者。
正则化(无论是显式损失惩罚还是隐式算法选择)是现代机器学习的核心。在数据稀缺、噪声或污染的情况下,控制模型复杂度尤为重要,这反映了对数据潜在结构的统计假设。本文研究高维对抗训练中正则化范数 $\lVert \cdot \rVert$ 的选择问题。我们首次精确推导出各类对抗攻击和正则化范数(包括非 $\ell_p$ 范数)下鲁棒正则化经验风险最小化器的渐近描述。同时,通过统一收敛分析,建立了该类问题的 Rademacher 复杂度上界。基于理论结果,我们定量刻画了扰动大小与最优 $\lVert \cdot \rVert$ 之间的关系,证实了在数据稀缺场景下,随着扰动增大,正则化范数的选择对对抗训练的影响愈发关键。
原文摘要 · Abstract (English)
Regularization, whether explicit in terms of a penalty in the loss or implicit in the choice of algorithm, is a cornerstone of modern machine learning. Indeed, controlling the complexity of the model class is particularly important when data is scarce, noisy or contaminated, as it translates a statistical belief on the underlying structure of the data. This work investigates the question of how to choose the regularization norm $\lVert \cdot \rVert$ in the context of high-dimensional adversarial training for binary classification. To this end, we first derive an exact asymptotic description of the robust, regularized empirical risk minimizer for various types of adversarial attacks and regularization norms (including non-$\ell_p$ norms). We complement this analysis with a uniform convergence analysis, deriving bounds on the Rademacher Complexity for this class of problems. Leveraging our theoretical results, we quantitatively characterize the relationship between perturbation size and the optimal choice of $\lVert \cdot \rVert$, confirming the intuition that, in the data scarce regime, the type of regularization becomes increasingly important for adversarial training as perturbations grow in size.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。