从统计角度设计新算法,缓解在线持续学习中的遗忘问题。
Statistical Theory of Multi-stage Newton Iteration Algorithm for Online Continual Learning
- 引入随机效应和无限参数维度,建立通用持续学习理论框架
- 提出多步牛顿迭代算法,降低矩阵求逆开销,提升计算效率
- 证明估计量渐近正态性,支持后续统计推断,适合研究者参考
我们聚焦在线持续学习中非平稳数据流的挑战,受限存储无法完整保留历史数据,导致序列任务训练时发生灾难性遗忘。为更有效地分析与解决该问题,我们从统计视角提出一种新框架,对所有模型参数引入随机效应,并允许参数维度发散至无穷大,形成持续学习问题的通用表述。为高效处理流式数据,我们开发了多步牛顿迭代算法,在特定场景下显著降低计算成本,缓解矩阵求逆负担。理论上,我们推导出估计量的渐近正态性,支持后续统计推断。通过合成数据实验及两个真实数据集分析,全面验证了所提方法的有效性。
原文摘要 · Abstract (English)
We focus on the critical challenge of handling non-stationary data streams in online continual learning environments, where constrained storage capacity prevents complete retention of historical data, leading to catastrophic forgetting during sequential task training. To more effectively analyze and address the problem of catastrophic forgetting in continual learning, we propose a novel continual learning framework from a statistical perspective. Our approach incorporates random effects across all model parameters and allows the dimension of parameters to diverge to infinity, offering a general formulation for continual learning problems. To efficiently process streaming data, we develop a Multi-step Newton Iteration algorithm that significantly reduces computational costs in certain scenarios by alleviating the burden of matrix inversion. Theoretically, we derive the asymptotic normality of the estimator, enabling subsequent statistical inference. Comprehensive validation through synthetic data experiments and two real datasets analyses demonstrates the effectiveness of our proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。