arXiv:2409.10965cs.CLcs.LG2024-09被引 11

对比单语与多语模型在非洲低资源语言间的迁移能力,发现多语模型表现更优。

Cross-lingual transfer of multilingual models on low resource African Languages

  • 比较单语与多语模型在卢旺达语与基隆迪语间的跨语言迁移效果
  • 微调后非洲语模型AfriBERT准确率达88.3%,优于其他模型
  • 适合关注非洲语言NLP、低资源语言迁移的研究者

大型多语言模型显著推动了自然语言处理研究,但其高资源需求和多元数据源带来的潜在偏见引发了对其在低资源语言中有效性的问题。相比之下,仅针对单一语言训练的单语模型可能更精准捕捉目标语言特征,从而提供更高准确性。本研究评估了从高资源语言向低资源语言(如卢旺达语和基隆迪语,均为班图语)迁移时,单语与多语模型的表现,涵盖基于Transformer的mBERT、AfriBERT、BantuBERTa,以及BiGRU、CNN、char-CNN等神经网络架构。模型在卢旺达语上训练并在基隆迪语上测试,通过微调评估性能提升与灾难性遗忘程度。结果显示,微调后AfriBERT达到最高跨语言准确率88.3%,而BiGRU作为神经模型表现最佳,准确率为83.3%。同时分析了微调后原语言的遗忘程度。尽管单语模型仍具竞争力,但本研究证明多语模型在资源受限场景下具备强大的跨语言迁移能力。

原文摘要 · Abstract (English)

Large multilingual models have significantly advanced natural language processing (NLP) research. However, their high resource demands and potential biases from diverse data sources have raised concerns about their effectiveness across low-resource languages. In contrast, monolingual models, trained on a single language, may better capture the nuances of the target language, potentially providing more accurate results. This study benchmarks the cross-lingual transfer capabilities from a high-resource language to a low-resource language for both, monolingual and multilingual models, focusing on Kinyarwanda and Kirundi, two Bantu languages. We evaluate the performance of transformer based architectures like Multilingual BERT (mBERT), AfriBERT, and BantuBERTa against neural-based architectures such as BiGRU, CNN, and char-CNN. The models were trained on Kinyarwanda and tested on Kirundi, with fine-tuning applied to assess the extent of performance improvement and catastrophic forgetting. AfriBERT achieved the highest cross-lingual accuracy of 88.3% after fine-tuning, while BiGRU emerged as the best-performing neural model with 83.3% accuracy. We also analyze the degree of forgetting in the original language post-fine-tuning. While monolingual models remain competitive, this study highlights that multilingual models offer strong cross-lingual transfer capabilities in resource limited settings.

多语模型低资源语言跨语言迁移非洲语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。