arXiv:2602.14078cs.LGcs.AI2026-02

用强化学习优化分类,缓解模型持续学习时的遗忘问题。

Policy Gradient with Adaptive Entropy Annealing for Continual Fine-Tuning

  • 将分类建模为马尔可夫决策过程,直接优化误分类率。
  • 在多个基准上优于传统交叉熵方法,提升适应能力。
  • 适合需要持续学习的视觉模型微调场景。

尽管大模型表现优异,但在类别增量设置下持续微调时仍易发生灾难性遗忘。参数高效微调(PEFT)通过限制可训练参数缓解此问题,但多数方法仍依赖交叉熵(CE)损失——作为0-1损失的近似。本文重新审视这一选择,从强化学习视角重拾真实目标(0-1损失)。通过将分类建模为单步马尔可夫决策过程,推导出期望策略梯度(EPG)方法,以低方差梯度估计直接最小化误分类误差。分析表明,CE可被解释为带有样本加权机制的EPG:CE通过强调低置信样本促进探索,而EPG则优先高置信样本。基于此,提出自适应熵退火(aEPG),一种从探索(类似CE)到利用(类似EPG)的渐进式学习策略。aEPG方法在多种基准和不同PEFT模块上均优于基于CE的方法。更广泛地,评估多种熵正则化方法后发现,降低输出预测分布的熵有助于提升预训练视觉模型的适应能力。

原文摘要 · Abstract (English)

Despite their success, large pretrained vision models remain vulnerable to catastrophic forgetting when adapted to new tasks in class-incremental settings. Parameter-efficient fine-tuning (PEFT) alleviates this by restricting trainable parameters, yet most approaches still rely on cross-entropy (CE) loss, a surrogate for the 0-1 loss, to learn from new data. We revisit this choice and revive the true objective (0-1 loss) through a reinforcement learning perspective. By formulating classification as a one-step Markov Decision Process, we derive an Expected Policy Gradient (EPG) method that directly minimizes misclassification error with a low-variance gradient estimation. Our analysis shows that CE can be interpreted as EPG with an additional sample-weighting mechanism: CE encourages exploration by emphasizing low-confidence samples, while EPG prioritizes high-confidence ones. Building on this insight, we propose adaptive entropy annealing (aEPG), a training strategy that transitions from exploratory (CE-like) to exploitative (EPG-like) learning. aEPG-based methods outperform CE-based methods across diverse benchmarks and with various PEFT modules. More broadly, we evaluate various entropy regularization methods and demonstrate that lower entropy of the output prediction distribution enhances adaptation in pretrained vision models.

持续学习强化学习模型微调熵正则

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。