提出高效可扩展的视觉变压器增量学习分类器,解决计算瓶颈问题。
Scalable Analytic Classifiers with Associative Drift Compensation for Class-Incremental Learning of Vision Transformers
- 用低秩分解降低分类器推理复杂度,提升大规模场景效率
- 在多个基准上达到当前最优性能,准确率显著优于基线方法
- 无需训练即可补偿特征漂移,适合长期增量学习任务
视觉变压器(ViTs)在类增量学习(CIL)中面临分类器重建阶段的严重计算瓶颈,现有方法多依赖高成本的迭代随机梯度下降(SGD)。我们发现解析正则化高斯判别分析(RGDA)可提供贝叶斯最优解,精度接近基于SGD的分类器,但其二次推断复杂度限制了在大规模场景中的应用。为此,我们提出低秩分解的RGDA(LR-RGDA),结合RGDA的表达能力与线性分类器的效率。通过利用协方差的低秩结构及Woodbury矩阵恒等式,将判别函数分解为全局仿射项与低秩二次扰动,将推断复杂度从$\mathcal{O}(Cd^2)$降至$\mathcal{O}(d^2 + Crd^2)$,其中$C$为类别数,$d$为特征维度,$r \ll d$为子空间秩。为缓解主干网络更新带来的表示漂移,进一步引入基于霍普菲尔德网络的分布补偿器(HopDC),该训练无关机制利用连续霍普菲尔德网络通过无标签锚点的关联记忆动态校准历史类别统计,并给出估计误差的理论界。在多个多样化CIL基准上的实验表明,本框架实现当前最优性能,为大规模视觉变压器增量学习提供了可扩展解决方案。
原文摘要 · Abstract (English)
Class-incremental learning (CIL) with Vision Transformers (ViTs) faces a major computational bottleneck during the classifier reconstruction phase, where most existing methods rely on costly iterative stochastic gradient descent (SGD). We observe that analytic Regularized Gaussian Discriminant Analysis (RGDA) provides a Bayes-optimal alternative with accuracy comparable to SGD-based classifiers; however, its quadratic inference complexity limits its use in large-scale CIL scenarios. To overcome this, we propose Low-Rank Factorized RGDA (LR-RGDA), a scalable classifier that combines RGDA's expressivity with the efficiency of linear classifiers. By exploiting the low-rank structure of the covariance via the Woodbury matrix identity, LR-RGDA decomposes the discriminant function into a global affine term refined by a low-rank quadratic perturbation, reducing the inference complexity from $\mathcal{O}(Cd^2)$ to $\mathcal{O}(d^2 + Crd^2)$, where $C$ is the class number, $d$ the feature dimension, and $r \ll d$ the subspace rank. To mitigate representation drift caused by backbone updates, we further introduce Hopfield-based Distribution Compensator (HopDC), a training-free mechanism that uses modern continuous Hopfield Networks to recalibrate historical class statistics through associative memory dynamics on unlabeled anchors, accompanied by a theoretical bound on the estimation error. Extensive experiments on diverse CIL benchmarks demonstrate that our framework achieves state-of-the-art performance, providing a scalable solution for large-scale class-incremental learning with ViTs. Code: https://github.com/raoxuan98-hash/lr_rgda_hopdc.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。