用词性切换和回放适配器缓解低资源语言建模中的遗忘问题。
Continual-learning for Modelling Low-Resource Languages from Large Language Models
- 基于词性设计代码切换策略，持续学习低资源语言。
- 在视觉问答和语言建模任务上实现稳定性能，避免灾难性遗忘。
- 适合低资源语言研究者，尤其关注多语言模型持续训练场景。
多语言场景下的语言模型构建面临诸多挑战，其中最突出的是灾难性遗忘。例如，通过微调大语言模型（LLM）构建的小型语言模型（SLM）在低资源语言上易出现遗忘问题。本文提出一种持续学习策略，结合词性（POS）引导的代码切换与回放适配器机制，在从大模型训练小模型的过程中有效缓解灾难性遗忘。在视觉问答和语言建模等视觉语言任务上的实验验证了该方法的有效性，表明其能保持对已有语言知识的稳定性并持续学习新语言。
原文摘要 · Abstract (English)
Modelling a language model for a multi-lingual scenario includes several potential challenges, among which catastrophic forgetting is the major challenge. For example, small language models (SLM) built for low-resource languages by adapting large language models (LLMs) pose the challenge of catastrophic forgetting. This work proposes to employ a continual learning strategy using parts-of-speech (POS)-based code-switching along with a replay adapter strategy to mitigate the identified gap of catastrophic forgetting while training SLM from LLM. Experiments conducted on vision language tasks such as visual question answering and language modelling task exhibits the success of the proposed architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。