为埃塞俄比亚语扩展词表，提升低资源语言模型表现
Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya
- 定制专用于阿姆哈拉语和提格雷尼亚语的分词器，扩充3万词元
- 在问答、命名实体识别等任务上显著超越XLM-R与Glot500
- 效果可迁移至非洲17种语言，适合低资源语言研究者
多语言预训练模型如XLM-R在主流语言上表现良好，但在低资源的盖兹文字语言中表现不佳，主要由于高未登录词率和拉丁字母分词器导致的过度子词切分。我们提出VEXMLM，一种针对阿姆哈拉语和提格雷尼亚语的XLM-R词汇扩展变体。通过在精选单语语料上训练语言特定的SentencePiece分词器，将XLM-R词表扩展3万盖兹文字子词，并通过子词平均初始化嵌入。VEXMLM采用两阶段训练：（1）在精选语料上继续进行掩码语言建模，（2）在问答、命名实体识别和情感分析任务上进行监督微调。VEXMLM在阿姆哈拉语和提格雷尼亚语的所有任务中均显著优于XLM-R和Glot500，尤其在未登录实体识别任务上提升明显。关键的是，其在阿姆哈拉语和提格雷尼亚语上的改进可迁移至非洲17种语言。结果表明，词表扩展与分词器适配是一种有效且计算高效的方法，可在不从头训练的前提下提升代表性不足语言的多语言模型性能。
原文摘要 · Abstract (English)
Multilingual pre-trained language models such as XLM-R perform well for major languages but struggle with low-resource Ge'ez-script languages, largely due to high out-of-vocabulary (OOV) rates and excessive subword fragmentation from Latin-script-centric tokenizers. We introduce VEXMLM, a vocabulary-extended variant of XLM-R targeting Amharic and Tigrinya. We train language-specific SentencePiece tokenizers on curated monolingual corpora, extend XLM-R's vocabulary with 30k Ge'ez-script subwords, and initialize their embeddings via subword averaging. VEXMLM undergoes two-stage training: (1) continued masked language modeling on the curated corpora and (2) supervised fine-tuning on question answering, named entity recognition, and sentiment analysis. VEXMLM substantially outperforms XLM-R and Glot500 across all evaluated tasks on Amharic and Tigrinya, with particularly strong gains on out-of-vocabulary entity recognition. Critically, improvements on Amharic and Tigrinya transfer to 17 languages in Africa. VEXMLM demonstrates that vocabulary expansion and tokenizer adaptation provide an effective, computationally efficient path to improve multilingual models for underrepresented languages without retraining from scratch. Resources: GitHub repository | Hugging Face models
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。