用能量模型视角解析对抗训练,发现过拟合与生成能力的内在联系。
Understanding Adversarial Training with Energy-based Models
- 从能量角度分析对抗样本与自然样本的差异,揭示过拟合机制。
- 提出Δ能量正则化,有效缓解灾难性过拟合与鲁棒过拟合。
- 无需显式训练生成,鲁棒分类器生成质量媲美混合模型。
本文借助能量模型(EBM)框架,深入理解分类器中的对抗训练(AT),并分析鲁棒分类器的内在生成能力。通过能量视角观察标准分类器,分析不同攻击生成的对抗样本与自然样本的能量差异。研究重点在于从能量角度解析对抗训练中的灾难性过拟合(CO)与鲁棒过拟合(RO)。我们发现,在发生CO或RO时,原始样本与其对抗样本之间的“能量差”(delta energy)行为显著发散。基于对能量动态及过拟合关系的分析,提出一种新的正则化方法——Δ能量正则化(DER),用于平滑训练过程中的能量景观。实验表明,DER在多个基准上有效缓解了CO和RO。进一步发现,将鲁棒分类器作为生成模型时,图像质量与多样性存在权衡。为此,提出基于局部类内主成分分析(PCA)与能量引导的改进生成技术,实现更优的类别初始化与自适应停止策略,显著提升样本多样性和生成质量。尽管未显式训练生成任务,其在Inception Score(IS)与Fréchet inception distance(FID)上仍达到与混合判别-生成模型相当的水平。
原文摘要 · Abstract (English)
We aim at using Energy-based Model (EBM) framework to better understand adversarial training (AT) in classifiers, and additionally to analyze the intrinsic generative capabilities of robust classifiers. By viewing standard classifiers through an energy lens, we begin by analyzing how the energies of adversarial examples, generated by various attacks, differ from those of the natural samples. The central focus of our work is to understand the critical phenomena of Catastrophic Overfitting (CO) and Robust Overfitting (RO) in AT from an energy perspective. We analyze the impact of existing AT approaches on the energy of samples during training and observe that the behavior of the ``delta energy' -- change in energy between original sample and its adversarial counterpart -- diverges significantly when CO or RO occurs. After a thorough analysis of these energy dynamics and their relationship with overfitting, we propose a novel regularizer, the Delta Energy Regularizer (DER), designed to smoothen the energy landscape during training. We demonstrate that DER is effective in mitigating both CO and RO across multiple benchmarks. We further show that robust classifiers, when being used as generative models, have limits in handling trade-off between image quality and variability. We propose an improved technique based on a local class-wise principal component analysis (PCA) and energy-based guidance for better class-specific initialization and adaptive stopping, enhancing sample diversity and generation quality. Considering that we do not explicitly train for generative modeling, we achieve a competitive Inception Score (IS) and Fréchet inception distance (FID) compared to hybrid discriminative-generative models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。