提出新方法平衡持续学习中旧知识保留与新知识获取的矛盾。
Exploring Stability-Plasticity Trade-offs for Continual Named Entity Recognition
- 从表征和权重两方面动态调节稳定性与可塑性。
- 在十组设置下优于现有方法,显著缓解遗忘问题。
- 适合需要持续更新实体类型的自然语言处理应用。
持续命名实体识别(CNER)旨在顺序更新模型以引入新实体类型。现有方法多依赖知识蒸馏(KD)保持旧知识,但导致模型过度稳定而难以学习新知识。为此,本文提出稳定性-可塑性权衡(SPT)方法,从表征和权重双角度平衡二者:表征层面,在原始KD中引入池化操作,降低表示维度以提升可塑性;权重层面,动态融合新旧模型参数,并通过权重引导的选择机制优先保留关键权重。此外,设计基于置信度的伪标签策略,利用旧模型预测非实体类型,缓解非实体语义漂移问题——这一挑战此前被忽视。在三个基准数据集上共十个CNER设置的实验表明,SPT方法显著优于现有方法,有效实现了稳定与可塑之间的良好权衡。
原文摘要 · Abstract (English)
Continual Named Entity Recognition (CNER) is an evolving field that focuses on sequentially updating an existing model to incorporate new entity types. Previous CNER methods primarily utilize Knowledge Distillation (KD) to preserve prior knowledge and overcome catastrophic forgetting, strictly ensuring that the representations of old and new models remain consistent. Consequently, they often impart the model with excessive stability (i.e., retention of old knowledge) but limited plasticity (i.e., acquisition of new knowledge). To address this issue, we propose a Stability-Plasticity Trade-off (SPT) method for CNER that balances these aspects from both representation and weight perspectives. From the representation perspective, we introduce a pooling operation into the original KD, permitting a level of plasticity by consolidating representation dimensions. From the weight perspective, we dynamically merge the weights of old and new models, strengthening old knowledge while maintaining new knowledge. During this fusion, we implement a weight-guided selective mechanism to prioritize significant weights. Moreover, we develop a confidence-based pseudo-labeling approach for the current non-entity type, which predicts entity types using the old model to handle the semantic shift of the non-entity type, a challenge specific to CNER that has largely been ignored by previous methods. Extensive experiments across ten CNER settings on three benchmark datasets demonstrate that our SPT method surpasses previous CNER approaches, highlighting its effectiveness in achieving a suitable stability-plasticity trade-off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。