压缩知识迁移提升持续学习推理效率,兼顾记忆与适应性。
Low-Complexity Inference in Continual Learning via Compressed Knowledge Transfer
- 用剪枝和知识蒸馏压缩大模型,分阶段优化推理复杂度。
- 在多个类增量学习基准上,精度与效率平衡优于主流方法。
- 适合对延迟和能耗敏感的实时持续学习场景。
持续学习(CL)旨在训练可逐步学习任务且不遗忘旧知识的模型。其核心挑战在于平衡稳定性(保持旧任务性能)与可塑性(适应新任务)。近期大预训练模型因兼具强泛化能力和抗遗忘性被广泛采用,但其高推理计算成本限制了实际应用,尤其在低延迟或低功耗场景中。为此,本文探索剪枝与知识蒸馏(KD)等模型压缩技术,提出两种针对类增量学习(CIL)的高效框架,其中任务类别信息在推理时不可见。剪枝框架采用训练阶段前、后剪枝策略实现分阶段压缩;知识蒸馏框架采用教师-学生架构,使大型预训练教师向紧凑学生模型传递下游相关知识。在多个CIL基准上的实验表明,所提框架在准确率与推理复杂度之间取得更优平衡,显著优于强基线。进一步分析两框架在准确率与效率间的权衡,为不同应用场景提供实践指导。
原文摘要 · Abstract (English)
Continual learning (CL) aims to train models that can learn a sequence of tasks without forgetting previously acquired knowledge. A core challenge in CL is balancing stability -- preserving performance on old tasks -- and plasticity -- adapting to new ones. Recently, large pre-trained models have been widely adopted in CL for their ability to support both, offering strong generalization for new tasks and resilience against forgetting. However, their high computational cost at inference time limits their practicality in real-world applications, especially those requiring low latency or energy efficiency. To address this issue, we explore model compression techniques, including pruning and knowledge distillation (KD), and propose two efficient frameworks tailored for class-incremental learning (CIL), a challenging CL setting where task identities are unavailable during inference. The pruning-based framework includes pre- and post-pruning strategies that apply compression at different training stages. The KD-based framework adopts a teacher-student architecture, where a large pre-trained teacher transfers downstream-relevant knowledge to a compact student. Extensive experiments on multiple CIL benchmarks demonstrate that the proposed frameworks achieve a better trade-off between accuracy and inference complexity, consistently outperforming strong baselines. We further analyze the trade-offs between the two frameworks in terms of accuracy and efficiency, offering insights into their use across different scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。