解决视觉Transformer在增量学习中特征分布漂移问题,提升模型长期性能。
Compensating Distribution Drifts in Class-incremental Learning of Pre-trained Vision Transformers
- 设计线性与弱非线性映射算子,对齐不同任务间特征分布。
- 结合知识蒸馏后,序列微调性能接近联合训练,在多个数据集上提升显著。
- 适合做预训练模型持续学习的研究者与工业应用开发者。
近期研究表明,对预训练视觉变压器(ViTs)进行序列微调(SeqFT),并利用类别特征近似分布进行分类器优化,是类增量学习(CIL)的有效策略。然而,该方法易受分布漂移影响,因共享主干参数的顺序优化导致先前学习类别的特征分布与更新模型不匹配,最终降低分类性能。为此,本文提出带漂移补偿的序列学习(SLDC),通过引入隐空间转换算子对齐任务间的特征分布。首先提出线性变体,通过正则化最小二乘求解特征前后映射;再扩展为弱非线性变体,假设理想转换介于线性与全非线性之间,采用可学习的弱非线性映射平衡灵活性与泛化能力。为进一步减少表示漂移,两种变体均结合知识蒸馏(KD)。在标准CIL基准上的大量实验表明,SLDC显著提升SeqFT性能;结合KD与SLDC后,其表现接近联合训练,在所有评估数据集上达到相似水平。代码已开源:https://github.com/raoxuan98-hash/sldc.git。
原文摘要 · Abstract (English)
Recent advances have shown that sequential fine-tuning (SeqFT) of pre-trained vision transformers (ViTs), followed by classifier refinement using approximate distributions of class features, can be an effective strategy for class-incremental learning (CIL). However, this approach is susceptible to distribution drift, caused by the sequential optimization of shared backbone parameters. This results in a mismatch between the distributions of the previously learned classes and that of the updater model, ultimately degrading the effectiveness of classifier performance over time. To address this issue, we introduce a latent space transition operator and propose Sequential Learning with Drift Compensation (SLDC). SLDC aims to align feature distributions across tasks to mitigate the impact of drift. First, we present a linear variant of SLDC, which learns a linear operator by solving a regularized least-squares problem that maps features before and after fine-tuning. Next, we extend this with a weakly nonlinear SLDC variant, which assumes that the ideal transition operator lies between purely linear and fully nonlinear transformations. This is implemented using learnable, weakly nonlinear mappings that balance flexibility and generalization. To further reduce representation drift, we apply knowledge distillation (KD) in both algorithmic variants. Extensive experiments on standard CIL benchmarks demonstrate that SLDC significantly improves the performance of SeqFT. Notably, by combining KD to address representation drift with SLDC to compensate distribution drift, SeqFT achieves performance comparable to joint training across all evaluated datasets. Code: https://github.com/raoxuan98-hash/sldc.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。