arXiv:2604.17578cs.LGmath.ST2026-04

提出任务依赖建模新方法,给出持续学习的可恢复性理论保证。

Recovery Guarantees for Continual Learning of Dependent Tasks: Memory, Data-Dependent Regularization, and Data-Dependent Weights

  • 假设当前任务数据是前序数据的非线性变换,显式建模任务间依赖关系。
  • 证明了经验回放、知识蒸馏等主流方法的估计误差上界,优于以往结果。
  • 适用于关注理论保障的算法设计者,尤其适合研究任务关联性场景。

持续学习(CL)旨在顺序学习多个任务而不遗忘先前知识。尽管近年已有大量实证进展,但其理论发展仍处于初期。核心挑战在于任务间数据分布变化,本文认为需理解这种分布间的依赖关系才能有效应对。为此,我们以非线性回归任务为模型,提出当前任务数据是前序数据的非线性变换这一假设,并在自然条件下,对多种实用的持续学习范式(如基于数据无关正则的经验回放、数据无关权重平衡损失、数据依赖权重回放,以及数据依赖正则化如知识蒸馏)给出了统计恢复保证(即估计误差的上界)。据我们所知,这些界在以往工作给出平凡结果的情况下仍具信息量。

原文摘要 · Abstract (English)

Continual learning (CL) is concerned with learning multiple tasks sequentially without forgetting previously learned tasks. Despite substantial empirical advances over recent years, the theoretical development of CL remains in its infancy. At the heart of developing CL theory lies the challenge that the data distribution varies across tasks, and we argue that properly addressing this challenge requires understanding this variation--dependency among tasks. To explicitly model task dependency, we consider nonlinear regression tasks and propose the assumption that these tasks are dependent in such a way that the data of the current task is a nonlinear transformation of previous data. With this model and under natural assumptions, we prove statistical recovery guarantees (more specifically, bounds on estimation errors) for several CL paradigms in practical use, including experience replay with data-independent regularization and data-independent weights that balance the losses of tasks, replay with data-dependent weights, and continual learning with data-dependent regularization (e.g., knowledge distillation). To the best of our knowledge, our bounds are informative in cases where prior work gives vacuous bounds.

持续学习理论分析任务依赖恢复保证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。