通过渐进式词表扩展，让大模型更高效学习阿拉伯语。
Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion
- 用类人类第二语言习得思路，逐步扩充阿拉伯语词表。
- 在多个阿拉伯语评测中表现媲美顶尖模型。
- 适合关注低资源语言大模型的开发者与研究者。
本文针对阿拉伯世界大语言模型发展滞后的问题，提出一种受人类第二语言（阿拉伯语）习得启发的渐进式词表扩展方法。传统使用阿拉伯语专用词表会导致训练初期大量词汇无法识别（OOV），影响知识学习。为此，我们改进BPE算法，在训练过程中动态扩展阿拉伯子词词表，使各阶段的未登录词比例保持平衡。消融实验证明该方法有效，AraLLaMA在多个阿拉伯语基准测试中表现优异，性能接近现有最佳阿拉伯语大模型。所有模型、训练数据、评估基准和代码均将开源。
原文摘要 · Abstract (English)
This paper addresses the critical need for democratizing large language models (LLM) in the Arab world, a region that has seen slower progress in developing models comparable to state-of-the-art offerings like GPT-4 or ChatGPT 3.5, due to a predominant focus on mainstream languages (e.g., English and Chinese). One practical objective for an Arabic LLM is to utilize an Arabic-specific vocabulary for the tokenizer that could speed up decoding. However, using a different vocabulary often leads to a degradation of learned knowledge since many words are initially out-of-vocabulary (OOV) when training starts. Inspired by the vocabulary learning during Second Language (Arabic) Acquisition for humans, the released AraLLaMA employs progressive vocabulary expansion, which is implemented by a modified BPE algorithm that progressively extends the Arabic subwords in its dynamic vocabulary during training, thereby balancing the OOV ratio at every stage. The ablation study demonstrated the effectiveness of Progressive Vocabulary Expansion. Moreover, AraLLaMA achieves decent performance comparable to the best Arabic LLMs across a variety of Arabic benchmarks. Models, training data, benchmarks, and codes will be all open-sourced.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。