arXiv:2605.03364cs.CV2026-05

解决长尾增量学习中遗忘与过拟合难题,提升小类识别能力。

Dynamic Distillation and Gradient Consistency for Robust Long-Tailed Incremental Learning

论文配图:Dynamic Distillation and Gradient Consistency for Robust Long-Tailed Incremental Learning
图 1 · 摘自论文原文
  • 用梯度一致性稳定训练过程,减少灾难性遗忘。
  • 动态调整知识蒸馏权重,最高提升5.0%准确率。
  • 适合处理类别不平衡的持续学习场景。

长尾类别增量学习(LT-CIL)旨在处理类别分布不均的数据集中的连续学习问题。该场景加剧了持续学习固有的灾难性遗忘问题,同时面临少数类学习不足和多数类过拟合的双重挑战。本文提出两种核心方法:首先引入梯度一致性正则化,利用梯度移动平均抑制剧烈波动,稳定训练过程;其次通过归一化熵衡量类别不平衡程度,动态调整知识蒸馏损失权重,实现旧知识保留与新信息学习之间的最优平衡。在CIFAR-100-LT、ImageNetSubset-LT和Food101-LT基准上的实验表明,该方法在多个设置下实现最高达5.0%的一致性准确率提升。尤其在从多数类到少数类的‘逆序’学习设置中表现显著,凸显其在不利学习动态下的鲁棒性。该框架在计算开销几乎不变的前提下达成性能提升,具有良好的实用性。

原文摘要 · Abstract (English)

The task of Long-tailed Class Incremental Learning (LT-CIL) addresses the sequential learning of new classes from datasets with imbalanced class distributions. This scenario intensifies the fundamental problem of catastrophic forgetting, inherent to continual learning, with the dual challenges of under-learning minority classes and overfitting majority classes. To tackle these combined issues, this paper proposes two main techniques. First, we introduce gradient consistency regularization, which leverages the moving average of gradients to suppress abrupt fluctuations and stabilize the training process. Second, we dynamically adjust the weight of the distillation loss by measuring the degree of class imbalance with normalized entropy. This adaptive weighting establishes an optimal balance between retaining old knowledge and acquiring new information. Experiments on the CIFAR-100-LT, ImageNetSubset-LT, and Food101-LT benchmarks show that our method achieves consistent accuracy improvements of up to 5.0\%. Furthermore, we demonstrate dramatic gains in the challenging 'In-ordered' setting, where tasks progress from majority to minority classes, highlighting our method's robustness in mitigating forgetting under unfavorable learning dynamics. This enhanced performance is achieved without a significant increase in computational overhead, demonstrating the practicality of our framework.

增量学习长尾分布知识蒸馏稳定性优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。