通过扩展词汇表让多语言模型支持更少资源语言
Effective vocabulary expansion of multilingual language models for extremely low-resource languages
- 用目标语言语料扩展词汇，再用双语词典初始化新词向量
- 在词性标注和命名实体识别任务上分别提升0.54%和2.60%
- 不降低原语言性能，对训练数据选择有强鲁棒性
多语言预训练语言模型（mPLMs）为众多低资源语言提供了显著优势。为了进一步拓展这些模型所支持的语言范围，许多工作聚焦于模型的持续预训练。然而，极少研究关注如何将mPLMs扩展至此前不支持的极低资源语言。为此，我们利用目标语言语料扩展模型词汇表，并筛选出原词汇表中偏向源语言（如英语）的子集，再借助双语词典初始化扩展词汇的表示。随后，基于这些新词汇表示，使用目标语言语料继续预训练mPLMs。实验结果表明，与使用随机初始化扩展词汇的基线方法相比，本文方法在词性标注（POS tagging）和命名实体识别（NER）任务上分别提升0.54%和2.60%。此外，该方法在不同训练语料选择下表现稳定，且持续预训练后源语言性能未下降。
原文摘要 · Abstract (English)
Multilingual pre-trained language models(mPLMs) offer significant benefits for many low-resource languages. To further expand the range of languages these models can support, many works focus on continued pre-training of these models. However, few works address how to extend mPLMs to low-resource languages that were previously unsupported. To tackle this issue, we expand the model's vocabulary using a target language corpus. We then screen out a subset from the model's original vocabulary, which is biased towards representing the source language(e.g. English), and utilize bilingual dictionaries to initialize the representations of the expanded vocabulary. Subsequently, we continue to pre-train the mPLMs using the target language corpus, based on the representations of these expanded vocabulary. Experimental results show that our proposed method outperforms the baseline, which uses randomly initialized expanded vocabulary for continued pre-training, in POS tagging and NER tasks, achieving improvements by 0.54% and 2.60%, respectively. Furthermore, our method demonstrates high robustness in selecting the training corpora, and the models' performance on the source language does not degrade after continued pre-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。