arXiv:2411.14202cs.LGcs.CV2024-11被引 1

通过相关性更新参数,减少存储并缓解持续学习中的灾难性遗忘。

Revised Regularization for Efficient Continual Learning through Correlation-Based Parameter Update in Bayesian Neural Networks

  • 用变分推断结合相关性正则化,动态调整参数分布。
  • 在多个数据集上优于现有方法,有效保留历史知识。
  • 适合资源受限场景下的持续学习应用。

我们提出一种基于贝叶斯神经网络的持续学习算法,采用变分推断,旨在克服现有方法的多个缺陷。在持续学习中,存储每一步的网络参数以保留知识面临挑战,尤其在无法访问历史数据的情况下,难以维持参数与数据集之间的对应关系。当前基于变分推断与KL散度的方法在节点不确定性更新时易引发灾难性遗忘,并导致部分节点耦合扰动。为此,我们提出:为降低全连接层参数存储开销,设计参数分布学习方法;在变分推断框架中引入针对参数均值与方差动态及分布的正则化项,兼顾KL散度优势并缓解其问题;引入重要性加权证据下界项,捕捉数据与参数的相关性,实现共性和特异性参数超空间基的存储。该方法将参数空间划分为共性和特性子空间,明确前后向知识迁移的有效条件,揭示网络参数与数据集的对应机制。实验表明,该方法在多种数据集及序列组合下表现优异,性能超越现有方法。

原文摘要 · Abstract (English)

We propose a Bayesian neural network-based continual learning algorithm using Variational Inference, aiming to overcome several drawbacks of existing methods. Specifically, in continual learning scenarios, storing network parameters at each step to retain knowledge poses challenges. This is compounded by the crucial need to mitigate catastrophic forgetting, particularly given the limited access to past datasets, which complicates maintaining correspondence between network parameters and datasets across all sessions. Current methods using Variational Inference with KL divergence risk catastrophic forgetting during uncertain node updates and coupled disruptions in certain nodes. To address these challenges, we propose the following strategies. To reduce the storage of the dense layer parameters, we propose a parameter distribution learning method that significantly reduces the storage requirements. In the continual learning framework employing variational inference, our study introduces a regularization term that specifically targets the dynamics and population of the mean and variance of the parameters. This term aims to retain the benefits of KL divergence while addressing related challenges. To ensure proper correspondence between network parameters and the data, our method introduces an importance-weighted Evidence Lower Bound term to capture data and parameter correlations. This enables storage of common and distinctive parameter hyperspace bases. The proposed method partitions the parameter space into common and distinctive subspaces, with conditions for effective backward and forward knowledge transfer, elucidating the network-parameter dataset correspondence. The experimental results demonstrate the effectiveness of our method across diverse datasets and various combinations of sequential datasets, yielding superior performance compared to existing approaches.

持续学习贝叶斯网络参数压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。