让神经网络自适应地扰动隐藏层表示,提升模型泛化能力。
Learning to Perturb Hidden Representations for Generalizable Deep Learning

- 设计LPA方法,通过PGD学习类别相关的激活扰动策略。
- 在多种任务上优于现有方法,尤其在长尾和域泛化场景中表现突出。
- 揭示了扩张/收缩扰动的正负增广作用,适用于需要鲁棒性的场景。
深度神经网络通过输入特征、隐藏激活、logits和损失构成的级联表示处理数据。尽管输入、logits和标签层面的扰动已被系统研究,但占据网络大部分计算量的中间隐藏激活却缺乏统一的扰动分析。本文建立了一个统一的隐藏激活扰动框架,发现Dropout、Manifold Mixup、对抗特征扰动等方法均施加特定形式的激活扰动,但多为无类别或随机策略。我们提出扩张扰动(增加激活范数)为正增广,收缩扰动(减少激活范数)为负增广,且扰动层决定其效果类似输入级增广(浅层)或logit级操控(深层)。提出学习扰动激活(LPA),在选定隐藏层上自适应地进行类别级扰动,基于PGD学习。进一步提供理论分析,揭示激活扰动与平坦极小值及层间扰动放大之间的联系。在平衡分类、长尾分类和域泛化任务上的实验表明,LPA持续优于现有方法,并与logit扰动方法如LPL具有互补优势。
原文摘要 · Abstract (English)
Deep neural networks process data through a cascade of representations: input features, hidden activations, logits, and loss. While perturbations at the input, logit, and label levels have been systematically studied, the intermediate hidden activations, which constitute the bulk of the network's computation, have received no unified perturbation analysis. In this paper, we establish a unified framework for hidden activation perturbation, revealing that Dropout, Manifold Mixup, adversarial feature perturbation, and related methods all impose specific forms of activation perturbation but with class-agnostic or random strategies. We conjecture that expansive perturbation (increasing activation norm) acts as positive augmentation, while contractive perturbation (decreasing activation norm) acts as negative augmentation, and that the perturbation layer determines whether the effect resembles input-level augmentation (shallow layers) or logit-level manipulation (deep layers). We propose Learning to Perturb Activations (LPA), which adaptively perturbs activations at a selected hidden layer with class-level perturbations learned via PGD. We further provide theoretical analysis connecting activation perturbation to flat minima and perturbation amplification through layers. Experiments on balanced classification, long-tail classification, and domain generalization demonstrate that LPA consistently outperforms existing methods and provides complementary benefits to logit perturbation methods such as LPL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。