通过约束神经元激活区间,高效防止持续学习中的遗忘问题。
InTAct: Interval-based Task Activation Consolidation for Continual Learning
- 在神经元层级定义任务专属激活区间,限制更新范围。
- 在多个基准上实现领先性能,且比参数约束更高效。
- 适合希望提升模型长期记忆能力的研究者和工程师。
持续学习是人工智能中的核心挑战,要求模型在学习新知识的同时保留旧知识。尽管已有多种方法取得进展,但多数缺乏对灾难性遗忘的严格数学保障。现有提供保障的方法多依赖参数空间的区间算术分析,如InterContiNet,但高维权重更新限制计算成本高昂。本文提出InTAct(基于区间的任务激活巩固),通过在神经元层级施加功能不变性来缓解灾难性遗忘。我们识别出前序任务对应的特定激活区间,并在此区域内约束参数更新,同时允许其他区域灵活适应。通过确保预测在嵌套激活区间内保持稳定,我们提供了可计算的函数不变性保障。相比参数空间约束,激活空间调控维度更低,效率更高。该方法与架构无关,适用于多种持续学习场景,尤其与提示学习结合后,在复杂基准上达到顶尖表现。
原文摘要 · Abstract (English)
Continual learning is a fundamental challenge in artificial intelligence that requires networks to acquire new knowledge while preserving previously learned representations. Despite the success of various approaches, most existing paradigms do not provide rigorous mathematical guarantees against catastrophic forgetting. Current methods that offer such guarantees primarily focus on analyzing the parameter space using \textit{interval arithmetic (IA)}, as seen in frameworks such as InterContiNet. However, restricting high-dimensional weight updates can be computationally expensive. In this work, we propose InTAct (Interval-based Task Activation Consolidation), a method that mitigates catastrophic forgetting by enforcing functional invariance at the neuron level. We identify specific activation intervals where previous tasks reside and constrain updates within these regions while allowing for flexible adaptation elsewhere. By ensuring that predictions remain stable within these nested activation intervals, we provide a tractable mathematical guarantee of functional invariance. We emphasize that regulating the activation space is significantly more efficient than parameter-based constraints, because the dimensionality of internal signals is much lower than that of the vast space of model weights. While our approach is architecture-agnostic and applicable to various continual learning settings, its integration with prompt-based methods enables it to achieve state-of-the-art performance on challenging benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。