揭示持续学习中模型容量动态变化规律，解释为何新任务越难学。
On Understanding of the Dynamics of Model Capacity in Continual Learning
- 提出有效模型容量（CLEMC）刻画稳定性与可塑性的动态平衡
- 证明无论架构或优化方法，模型表征新任务能力随分布变化而下降
- 覆盖从小型网络到大语言模型的多类模型，理论与实验结合
持续学习中的稳定性-可塑性困境与神经网络的容量密切相关——即其表征任务的能力。本文引入持续学习有效模型容量（CLEMC），用于刻画稳定性与可塑性平衡点的动态行为。我们建立一个差分方程，模拟神经网络、任务数据与优化过程之间的相互作用。通过该模型，我们证明：有效容量及其对应的稳定性-可塑性平衡点本质上是非平稳的。无论采用何种网络架构或优化方法，当新任务分布与先前任务差异较大时，模型表征新任务的能力均会下降。我们通过大量实验验证了上述理论结论，涵盖从小型前馈网络、卷积网络到中等规模图神经网络及含数百万参数的基于Transformer的大语言模型。
原文摘要 · Abstract (English)
The stability-plasticity dilemma, closely related to a neural network's (NN) capacity-its ability to represent tasks-is a fundamental challenge in continual learning (CL). Within this context, we introduce CL's effective model capacity (CLEMC) that characterizes the dynamic behavior of the stability-plasticity balance point. We develop a difference equation to model the evolution of the interplay between the NN, task data, and optimization procedure. We then leverage CLEMC to demonstrate that the effective capacity-and, by extension, the stability-plasticity balance point is inherently non-stationary. We show that regardless of the NN architecture or optimization method, a NN's ability to represent new tasks diminishes when incoming task distributions differ from previous ones. We conduct extensive experiments to support our theoretical findings, spanning a range of architectures-from small feedforward network and convolutional networks to medium-sized graph neural networks and transformer-based large language models with millions of parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。