用可学习软标签蒸馏记忆数据，缓解持续学习中的遗忘问题。
Data-Distill-Net: A Data Distillation Approach Tailored for Reply-based Continual Learning
- 构建可学习的记忆缓冲区，通过软标签蒸馏全局信息。
- 在多个数据集上显著减少遗忘，性能媲美先进方法。
- 轻量级设计避免过拟合，适合资源受限的持续学习场景。
基于重放的持续学习方法假设小规模子集训练模型能有效最小化完整数据集的经验风险。这类方法通过维护一个存储先前任务采样数据的内存缓冲区来巩固旧知识。然而，由于内存缓冲区容量有限及数据选择采用启发式策略，该假设在实践中难以保证。为此，我们提出一种专为持续学习设计的新数据蒸馏框架，通过可学习的内存缓冲区，从当前任务数据和历史记忆中蒸馏全局信息。为避免参数化整个缓冲区带来的计算开销与过拟合风险，引入轻量级蒸馏模块，仅通过生成可学习的软标签实现记忆数据的全局信息蒸馏。大量实验表明，该方法在多种数据集上均能取得有竞争力的性能，并有效缓解遗忘现象。源代码将公开。
原文摘要 · Abstract (English)
Replay-based continual learning (CL) methods assume that models trained on a small subset can also effectively minimize the empirical risk of the complete dataset. These methods maintain a memory buffer that stores a sampled subset of data from previous tasks to consolidate past knowledge. However, this assumption is not guaranteed in practice due to the limited capacity of the memory buffer and the heuristic criteria used for buffer data selection. To address this issue, we propose a new dataset distillation framework tailored for CL, which maintains a learnable memory buffer to distill the global information from the current task data and accumulated knowledge preserved in the previous memory buffer. Moreover, to avoid the computational overhead and overfitting risks associated with parameterizing the entire buffer during distillation, we introduce a lightweight distillation module that can achieve global information distillation solely by generating learnable soft labels for the memory buffer data. Extensive experiments show that, our method can achieve competitive results and effectively mitigates forgetting across various datasets. The source code will be publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。