用模块化方法微调预训练模型，提升低资源语言表现
Modular Monolingual Adaptation using Pretrained Language Models
- 替换目标语言词元，冻结对应嵌入，仅微调其余部分
- 在苏格兰盖尔语等语言上，任务准确率提升10%以上
- 适合资源极少的语言，如仅有8500条数据的克丘亚语
为低资源语言构建单语语言模型通常依赖于在目标语言上全模型微调预训练语言模型（PLM），该方法能有效迁移知识。已有研究显示，使用语言特定分词器可增强适应性。本文提出，全模型微调常非必要，提出更模块化的方法：替换词元，冻结对应嵌入，仅微调其余部分。实验采用苏格兰盖尔语、爱尔兰语和克丘亚语，其中克丘亚语为极低资源语言（仅8500条训练样本）。在自然语言理解任务（掩码填充、命名实体识别、词性标注）上，新方法显著提升性能。同时，系统分析了训练策略、预训练嵌入选择与模型结构的影响。
原文摘要 · Abstract (English)
Building monolingual language models (LMs) for low-resource languages typically relies on adapting pretrained language models (PLMs) by finetuning the whole model on the target language. This approach is widely favored over training from scratch, as it enables effective knowledge transfer. Additionally, prior work has shown that using a language-specific tokenizer can enhance the adaptability. In this work, we hypothesize that full model tuning is often unnecessary and propose a more modular approach. Specifically, we replace the tokens, freeze the corresponding embeddings, and tune the rest of the model. We use Scottish Gaelic, Irish, and Quechua for our experiments, with Quechua being a very low-resource language (8.5k training instances). Evaluation on natural language understanding (NLU) tasks -- mask filling, NER, and POS -- shows that our proposed approach improves performance when adapting models to low-resource languages. Additionally, we provide a comprehensive analysis of the effectiveness of training strategies, the choice of pretrained embeddings, and models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。