提出分层自适应正则化,有效缓解持续学习中的遗忘问题。
Layers Matter: Why Continual Learning Regularization Should Be Layer-Adaptive
- 基于层级主曲率设计正则化策略,优于传统参数级方法。
- 实验显示同均值的层间主特征值差异可达层宽量级。
- 适合关注模型长期稳定性的研究者与工业部署场景。
持续学习正则化方法如EWC通过惩罚参数变化来防止遗忘,通常使用逐参数重要性(如对角Fisher值)。虽然逐参数更灵活,但每层的对角Fisher仅是其实际曲率的弱总结,忽略了控制遗忘的关键——最高特征值信息。对抗位翻转攻击和海森谱分析表明,神经网络中这一缺失的层级敏感度可跨越数量级。在块对角海森假设下,我们证明三点:遗忘可分解为按各层最高海森特征值加权的层级项之和;对角Fisher权重无法恢复该特征值;例如两层平均Fisher相同,但最高特征值可能相差达层宽量级。相同遗忘水平下,统一正则化会使新任务性能下降,降幅与层条件数成正比。理论推导出简单方案:强保护浅层,允许深层自由调整。将此应用于EWC与SLCA,在平均性能与遗忘指标上均有显著提升。
原文摘要 · Abstract (English)
Continual learning regularizers like EWC fight forgetting by penalizing changes from previous-task parameters with per-parameter importance, typically diagonal Fisher values. Per-parameter looks more flexible than per-layer, but each layer's diagonal Fisher is a weak summary of its actual curvature, missing the top-eigenvalue information that controls forgetting. Adversarial bit-flip attacks and Hessian-spectrum studies show that this missing per-layer sensitivity spans orders of magnitude in neural networks. Under a block-diagonal Hessian assumption, the layer-level analogue of EWC's existing diagonal assumption, we prove three things. Forgetting decomposes as a sum of per-layer terms weighted by each layer's top Hessian eigenvalue. Diagonal-Fisher weights cannot recover this eigenvalue. For instance, two layers with identical Fisher averages can have top eigenvalues differing by a factor as large as the layer width. For the same level of forgetting, uniform regularization loses new-task performance by an amount scaling with the layer condition number. Our theoretical analysis leads to a simple recipe: protect early layers strongly, let deeper layers move. We apply this recipe to EWC and SLCA and show clear improvements in average performance and forgetting metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。