提出可学习小波激活函数,解决持续学习中遗忘与泛化难题。
Sustaining Plasticity via Learnable Wavelet Activations in Continual Learning

- 用小波分解激活函数,分离高低频成分,对抗频谱偏差。
- 动态注入机制提升新任务适应力,正则化保障旧知识稳定。
- 理论证明架构必要性,适合需要长期学习的场景。
持续学习中的塑性丧失已成为关键挑战,严重影响模型对连续任务的学习能力。现有固定形式的激活函数存在对低频变化的固有频谱偏差,而可学习变体又因更新无约束导致灾难性遗忘。为此,本文提出一种新型可学习小波激活函数,将激活函数分解为低频与高频成分,显式缓解频谱偏差。同时,采用动态小波注入机制自适应增强新任务的塑性,并引入正则化策略确保已有知识的稳定性。理论上,本文严格证明了混合小波架构在高效$L^2$逼近中的结构必要性,并表明解耦学习率机制可有效恢复网络对高频信息的塑性。此外,还推导出基于损失驱动的注入触发机制,精确引导注入时机。大量实证评估表明,该方法在整个学习过程中保持优异的可训练性与泛化性能,在多个持续学习基准上达到当前最优表现。
原文摘要 · Abstract (English)
Plasticity loss has emerged as a critical challenge in continual learning that significantly hinders the acquisition of sequential tasks. While optimizing activation designs offers a potential solution, current fixed-form functions suffer from an inherent spectral bias towards low-frequency variations, whereas learnable variants permit unconstrained updates that induce catastrophic forgetting. To address these limitations, we propose a novel learnable wavelet activation that decomposes the activation function into low-frequency and high-frequency components to explicitly counter spectral bias. Furthermore, we employ dynamic wavelet injection to adaptively enhance plasticity for new tasks, alongside a regularization strategy to ensure the stability of previous learned knowledge. Theoretically, we provide rigorous mathematical guarantees for the proposed framework, proving the structural necessity of the hybrid wavelet architecture for efficient $L^2$ approximation and demonstrating that the decoupled learning rate mechanism successfully restores network plasticity for high-frequency information. Additionally, we provide a formal derivation of the loss-driven injection trigger mechanism to precisely guide the injection. Extensive empirical evaluations demonstrate that our approach maintains superior trainability and generalization throughout the learning process and achieves state-of-the-art performance across diverse continual learning benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。