为小语种定制分词器，让大模型更公平地支持英语和非拉丁语
From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages
- 为小语种扩展词表并初始化新嵌入，低成本训练双语大模型
- 在乌俄、阿拉伯、格鲁吉亚语上提升生成质量，减少语言混淆
- 提出新评估指标，证明词表大小直接影响生成文本质量
本文提出一种模型无关的低成本方法，用于构建支持英语和任意目标语言的双语基础大语言模型。该方法包括词表扩展、新嵌入初始化、模型训练与评估。我们在三种使用非拉丁字母的语言（乌克兰语、阿拉伯语、格鲁吉亚语）上进行实验。结果表明，该方法在提升语言性能的同时降低了计算成本，缓解了对低资源语言的不公惩罚，减少了代码切换和语法断裂等不良现象。此外，我们引入新评估指标，发现词表大小显著影响生成文本质量。
原文摘要 · Abstract (English)
In this paper, we propose a model-agnostic cost-effective approach to developing bilingual base large language models (LLMs) to support English and any target language. The method includes vocabulary expansion, initialization of new embeddings, model training and evaluation. We performed our experiments with three languages, each using a non-Latin script - Ukrainian, Arabic, and Georgian. Our approach demonstrates improved language performance while reducing computational costs. It mitigates the disproportionate penalization of underrepresented languages, promoting fairness and minimizing adverse phenomena such as code-switching and broken grammar. Additionally, we introduce new metrics to evaluate language quality, revealing that vocabulary size significantly impacts the quality of generated text.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。