用通用分词器提升大模型跨语言适应能力，效果显著且代价低。
One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers
- 用覆盖更多语言的通用分词器，增强模型后期扩展新语言的能力。
- 相比专用分词器，新语言适配成功率提升最高达20.2%。
- 对从未见过的语言也有5%的性能增益，适合多语言研究者使用。
为多种语言同时预训练大规模多语言大模型面临模型容量有限、高质量数据稀缺和计算资源不足的挑战。此外，分词器的语言覆盖不足，使得仅通过后训练阶段补充新语言更加困难。本文研究早期训练中低成本干预如何提升模型的‘语言可塑性’（即训练后对新语言的适应能力）。我们聚焦于分词器设计，提出使用覆盖语言数量超过主预训练语言的通用分词器，以实现预训练后高效扩展语言支持。在多种语言组和不同训练策略下的系统实验表明，通用分词器能显著提升语言适应能力，新语言适配胜率最高提升20.2%。此外，对于分词器和预训练中均未出现的语言，也可实现最高5%的胜率提升。该方法在大幅扩展语言覆盖的同时，对原预训练语言的性能影响极小。
原文摘要 · Abstract (English)
Pretraining massively multilingual Large Language Models (LLMs) for many languages at once is challenging due to limited model capacity, scarce high-quality data, and compute constraints. Moreover, the lack of language coverage of the tokenizer makes it harder to address the gap for new languages purely at the post-training stage. In this work, we study what relatively cheap interventions early on in training improve "language plasticity", or adaptation capabilities of the model post-training to new languages. We focus on tokenizer design and propose using a universal tokenizer that is trained for more languages than the primary pretraining languages to enable efficient adaptation in expanding language coverage after pretraining. Our systematic experiments across diverse groups of languages and different training strategies show that a universal tokenizer enables significantly higher language adaptation, with up to 20.2% increase in win rates compared to tokenizers specific to pretraining languages. Furthermore, a universal tokenizer also leads to better plasticity towards languages that are completely unseen in the tokenizer and pretraining, by up to 5% win rate gain. We achieve this adaptation to an expanded set of languages with minimal compromise in performance on the majority of languages included in pretraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。