用离散键值瓶颈实现小模型高效持续学习，防遗忘且省算力。
Efficient Continual Learning for Small Language Models with a Discrete Key-Value Bottleneck
- 引入离散键值瓶颈，仅更新局部参数防止遗忘。
- 在4种场景下表现优于主流方法，计算成本更低。
- 无需任务标识，在挑战性场景中仍有效，适合资源受限部署。
持续学习在自然语言处理中仍具挑战，因新数据训练易导致先前知识的灾难性遗忘。本文针对编码器-仅限语言模型提出离散键值瓶颈（DKVB），通过局部化更新实现高效持续学习。受视觉领域离散键值瓶颈启发，结合NLP特有挑战，对比多种瓶颈结构，并提出一种新的、与任务无关的离散键初始化方法。在四个持续学习场景中评估DKVB，结果表明其能有效缓解灾难性遗忘。实验显示，该方法性能媲美主流持续学习技术，同时计算开销更低。此外，即使在无任务标识的单头持续学习场景中，DKVB仍保持有效性。
原文摘要 · Abstract (English)
Continual learning remains a challenge across various natural language processing (NLP) tasks, as models updated with new training data often risk catastrophic forgetting of previously acquired knowledge. We introduce a discrete key-value bottleneck (DKVB) for encoder-only language models, enabling efficient continual learning through localized updates. Inspired by a discrete key-value bottleneck in vision, we consider new and NLP-specific challenges. We compare different bottleneck architectures for NLP and introduce a new, task-independent initialization technique for the discrete keys. We evaluate our DKVB for NLP in four continual learning scenarios and show that it alleviates catastrophic forgetting. Our experiments demonstrate that the proposed approach achieves competitive performance compared to popular continual learning methods while incurring lower computational costs. Furthermore, we show that DKVB remains effective even in challenging single-head continual learning scenarios where no task ID is provided.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。