用统一资源预算评估在线持续学习，提升效率与性能
Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling
- 以FLOPs和内存字节数定义统一资源预算
- 自适应层冻结减少计算量，损失精度极小
- 基于频率采样提升记忆检索效率，加速学习
多数在线持续学习（CL）方法采用单轮训练，并限制重放缓存大小。然而，单轮训练导致不同CL算法的计算量差异显著，且存储日志或模型所增加的内存成本常被忽视。为避免计算与存储预算不一致影响实际比较，本文提出以浮点运算量（FLOPs）和总内存大小（字节）作为计算与内存预算的衡量标准，实现同资源预算下的算法对比与开发。在有限总预算下，本文提出自适应层冻结策略，对信息量少的批次不更新对应层，从而降低计算开销且精度损失可忽略；同时提出一种基于频率的内存检索方法，使模型在更少迭代次数内获得与随机检索相当的知识。在CIFAR-10/100、CLEAR-10/100及ImageNet-1K数据集上的实证结果表明，该方法在相同总预算下优于现有最先进方法。
原文摘要 · Abstract (English)
The majority of online continual learning (CL) advocates single-epoch training and imposes restrictions on the size of replay memory. However, single-epoch training would incur a different amount of computations per CL algorithm, and the additional storage cost to store logit or model in addition to replay memory is largely ignored in calculating the storage budget. Arguing different computational and storage budgets hinder fair comparison among CL algorithms in practice, we propose to use floating point operations (FLOPs) and total memory size in Byte as a metric for computational and memory budgets, respectively, to compare and develop CL algorithms in the same 'total resource budget.' To improve a CL method in a limited total budget, we propose adaptive layer freezing that does not update the layers for less informative batches to reduce computational costs with a negligible loss of accuracy. In addition, we propose a memory retrieval method that allows the model to learn the same amount of knowledge as using random retrieval in fewer iterations. Empirical validations on the CIFAR-10/100, CLEAR-10/100, and ImageNet-1K datasets demonstrate that the proposed approach outperforms the state-of-the-art methods within the same total budget
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。