用全局工作空间模型提升在线增量学习的稳定与适应性
Multi-level Collaborative Distillation Meets Global Workspace Model: A Unified Framework for OCIL
- 构建共享隐式记忆GWM,动态整合多学生模型参数
- 在三个基准上验证,不同内存预算下性能显著优于现有方法
- 适合资源受限场景下的持续学习,尤其适合需要长期记忆的系统
在线类别增量学习(OCIL)使模型能从非独立同分布的数据流中持续学习,相比离线学习更贴近真实场景。由于数据仅可访问一次,对模型稳定性与可塑性的平衡提出更高要求。在实际应用中内存缓冲区受限,传统基于回放的方法效果下降,而集成方法虽提升可塑性却常牺牲稳定性。受全局工作空间理论(GWT)启发,本文提出一种新方法:通过全局工作空间模型(GWM)构建共享隐式记忆,融合每批次所有学生模型参数,捕捉历史学习轨迹并作为知识巩固的动态锚点。类似GWT的广播机制,GWM定期重分配给各学生模型,稳定学习过程并促进跨任务一致性。同时引入多层次协同蒸馏机制,强制学生间互相对齐,并将每个学生与GWM对齐,从而在保持对新任务适应性的同时保留历史知识。在三个标准OCIL基准上的实验表明,该方法在多种内存预算下显著提升多个OCIL模型性能。代码已公开于https://github.com/susususushi/GWM。
原文摘要 · Abstract (English)
Online Class-Incremental Learning (OCIL) enables models to learn continuously from non-i.i.d. data streams. Since samples of the data streams can be seen only once, it is more suitable for real-world scenarios compared to offline learning. However, this constraint intensifies the challenge for OCIL in maintaining an appropriate balance between stability and plasticity. Moreover, under stricter memory buffer constraints in real world, current replay-based methods are less effective. While ensemble methods improve plasticity, they often struggle with stability. Inspired by the Global Workspace Theory (GWT), we propose a novel approach that enhances ensemble learning through a Global Workspace Model (GWM)-a shared, implicit memory that guides the learning of multiple student models. The GWM is formed by fusing the parameters of all students within each training batch, capturing the historical learning trajectory and serving as a dynamic anchor for knowledge consolidation. Like the broadcasting mechanism of GWT, the GWM is redistributed periodically to students, stabilizing learning and promoting cross-task consistency. In addition, we introduce a multi-level collaborative distillation mechanism. It enforces peer-to-peer consistency among students and preserves historical knowledge by aligning each student with the GWM. As a result, student models remain adaptable to new tasks while maintaining previously learned knowledge, striking a better balance between stability and plasticity. Extensive experiments on three standard OCIL benchmarks show that our method delivers significant performance improvement for several OCIL models across various memory budgets. The code is available at https://github.com/susususushi/GWM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。