arXiv:2512.03537cs.LGstat.ML2025-12

轻量插件提升持续学习分类准确率,干扰小效率高

Pushing the Limits of Distillation-Based Continual Learning via Classifier-Proximal Lightweight Plugins

  • 在分类器附近插入轻量残差模块,实现语义级修正
  • 大模型基准上提升8%准确率,主干参数仅增4%
  • 兼容其他插件方法,适合高效持续学习场景

持续学习要求模型在不断变化的数据流中持续学习并保留旧知识。基于知识蒸馏的方法在共享单模型框架下以低存储开销保留历史知识,但仍受限于稳定性-可塑性困境:知识获取与保留通过耦合目标优化,现有增强方法未突破此根本瓶颈。为此,我们提出一种名为蒸馏感知轻量组件(DLC)的插件扩展范式。DLC将轻量残差插件部署于基础特征提取器的分类器邻近层,实现语义级残差修正,提升分类精度同时最小化对整体特征提取过程的干扰。推理时,聚合插件增强的表示生成分类预测。为缓解非目标插件的干扰,引入轻量加权单元,自动学习不同插件表示的重要性分数。DLC在大规模基准上实现8%的准确率提升,主干参数仅增加4%,展现出极佳效率。此外,DLC可与其他即插即用的持续学习增强方法兼容,并在组合使用时带来额外增益。

原文摘要 · Abstract (English)

Continual learning requires models to learn continuously while preserving prior knowledge under evolving data streams. Distillation-based methods are appealing for retaining past knowledge in a shared single-model framework with low storage overhead. However, they remain constrained by the stability-plasticity dilemma: knowledge acquisition and preservation are still optimized through coupled objectives, and existing enhancement methods do not alter this underlying bottleneck. To address this issue, we propose a plugin extension paradigm termed Distillation-aware Lightweight Components (DLC) for distillation-based CL. DLC deploys lightweight residual plugins into the base feature extractor's classifier-proximal layer, enabling semantic-level residual correction for better classification accuracy while minimizing disruption to the overall feature extraction process. During inference, plugin-enhanced representations are aggregated to produce classification predictions. To mitigate interference from non-target plugins, we further introduce a lightweight weighting unit that learns to assign importance scores to different plugin-enhanced representations. DLC could deliver a significant 8% accuracy gain on large-scale benchmarks while introducing only a 4% increase in backbone parameters, highlighting its exceptional efficiency. Moreover, DLC is compatible with other plug-and-play CL enhancements and delivers additional gains when combined with them.

持续学习轻量插件知识蒸馏分类器修正

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。