针对长尾数据下对抗训练表现不佳的问题,提出自蒸馏新方法提升小类鲁棒性。
Long-tailed Adversarial Training with Self-Distillation
- 用平衡数据训练自教师模型,指导原模型学习
- 在CIFAR-10等数据集上尾类抗攻击准确率提升超20个百分点
- 方法简单有效,适合长尾分布下的鲁棒模型训练
对抗训练显著提升模型对抗鲁棒性,但其性能主要集中在均衡数据集上。在非均衡或长尾分布场景中,对抗鲁棒性更难实现,主要因尾部类别样本稀少。现有研究多将传统长尾自然训练与已有鲁棒性方法结合,未能有效解决核心问题。本文深入分析了对抗训练在长尾分布中对尾类表现差的原因,并提出一种新颖的自蒸馏方法,通过使用从原始长尾数据集中采样的平衡数据训练一个平衡的自教师模型,来增强主模型在长尾分布下的对抗鲁棒性。大量实验表明,该方法在干净准确率和鲁棒准确率方面均达到当前最优水平,在多个数据集上显著提升尾类性能:在CIFAR-10、CIFAR-100和Tiny-ImageNet上,对抗PGD攻击时尾类准确率分别提升20.3、7.1和3.8个百分点,同时获得最高鲁棒准确率。
原文摘要 · Abstract (English)
Adversarial training significantly enhances adversarial robustness, yet superior performance is predominantly achieved on balanced datasets. Addressing adversarial robustness in the context of unbalanced or long-tailed distributions is considerably more challenging, mainly due to the scarcity of tail data instances. Previous research on adversarial robustness within long-tailed distributions has primarily focused on combining traditional long-tailed natural training with existing adversarial robustness methods. In this study, we provide an in-depth analysis for the challenge that adversarial training struggles to achieve high performance on tail classes in long-tailed distributions. Furthermore, we propose a simple yet effective solution to advance adversarial robustness on long-tailed distributions through a novel self-distillation technique. Specifically, this approach leverages a balanced self-teacher model, which is trained using a balanced dataset sampled from the original long-tailed dataset. Our extensive experiments demonstrate state-of-the-art performance in both clean and robust accuracy for long-tailed adversarial robustness, with significant improvements in tail class performance on various datasets. We improve the accuracy against PGD attacks for tail classes by 20.3, 7.1, and 3.8 percentage points on CIFAR-10, CIFAR-100, and Tiny-ImageNet, respectively, while achieving the highest robust accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。