arXiv:2603.18641cs.CL2026-03

对比多种方法缓解语言模型持续学习中的遗忘问题,发现重放策略最有效。

A Comparative Empirical Study of Catastrophic Forgetting Mitigation in Sequential Task Adaptation for Continual Natural Language Processing Systems

  • 构建10个不重叠意图分类任务,测试三种模型在不同持续学习策略下的表现
  • 仅用重放(MIR)的策略效果最好,组合使用时性能更优且可正向迁移
  • 模型架构与持续学习方法需协同选择,无万能方案

实际应用中的神经语言模型需持续适应新任务与领域而不遗忘旧知识。本文针对持续意图分类,基于CLINC150数据集构建了10个标签不重叠的任务序列,评估了前馈神经网络(ANN)、门控循环单元(GRU)和Transformer编码器三种骨干架构,在多种持续学习(CL)策略下的表现。涵盖三类代表性方法:基于重放的极大干扰检索(MIR)、基于正则化的无遗忘学习(LwF),以及通过硬注意力隔离参数的HAT,分别单独及两两、三者组合使用。评估指标包括平均准确率、宏平均F1和反向迁移,以衡量稳定性与灵活性的权衡。结果表明,直接顺序微调对所有架构均导致严重遗忘,单一方法无法完全避免。重放是关键:MIR作为独立策略最可靠,包含重放的组合(如MIR+HAT、MIR+LwF、MIR+LwF+HAT)始终取得高最终性能并实现近零或轻微正向反向迁移。最优配置依赖于架构:对ANN和Transformer,MIR+HAT最佳;对GRU,MIR+LwF+HAT表现最优,且某些情况下优于联合训练,体现正则化效应。研究强调设计持续意图分类系统时需协同选择模型架构与持续学习机制。

原文摘要 · Abstract (English)

Neural language models deployed in real-world applications must continually adapt to new tasks and domains without forgetting previously acquired knowledge. This work presents a comparative empirical study of catastrophic forgetting mitigation in continual intent classification. Using the CLINC150 dataset, we construct a 10-task label-disjoint scenario and evaluate three backbone architectures: a feed-forward Artificial Neural Network (ANN), a Gated Recurrent Unit (GRU), and a Transformer encoder, under a range of continual learning (CL) strategies. We consider one representative method from each major CL family: replay-based Maximally Interfered Retrieval (MIR), regularization-based Learning without Forgetting (LwF), and parameter-isolation via Hard Attention to Task (HAT), both individually and in all pairwise and triple combinations. Performance is assessed with average accuracy, macro F1, and backward transfer, capturing the stability-plasticity trade-off across the task sequence. Our results show that naive sequential fine-tuning suffers from severe forgetting for all architectures and that no single CL method fully prevents it. Replay emerges as a key ingredient: MIR is the most reliable individual strategy, and combinations that include replay (MIR+HAT, MIR+LwF, MIR+LwF+HAT) consistently achieve high final performance with near-zero or mildly positive backward transfer. The optimal configuration is architecture-dependent. MIR+HAT yields the best result for ANN and Transformer, MIR+LwF+HAT, on the other hand, works the best for GRU, and in several cases CL methods even surpass joint training, indicating a regularization effect. These findings highlight the importance of jointly selecting backbone architecture and CL mechanism when designing continual intent-classification systems.

持续学习意图识别遗忘缓解模型架构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。