对比11种南非语言的N-gram与预训练模型，发现Serengeti表现最佳。
From N-grams to Pre-trained Multilingual Models For Language Identification
- 用N-gram和多语言大模型做语言识别，强调数据量对频率分布的影响
- Serengeti在各类模型中平均表现最优，优于mBERT、AfriBERTa等
- 提出轻量级za_BERT_lid模型，性能接近顶尖非洲特化模型
本文研究了N-gram模型与大规模预训练多语言模型在11种南非语言上的语言识别（LID）性能。对于N-gram模型，研究发现有效数据量选择对构建目标语言的频率分布至关重要，能提升语言排序效果。针对预训练多语言模型，实验涵盖mBERT、RemBERT、XLM-r及非洲特化模型AfriBERTa、Afro-XLMr、AfroLM和Serengeti。同时与CLD V3、AfroLID、GlotLID、OpenLID等主流工具对比，凸显聚焦式识别的重要性。结果表明，Serengeti在从N-gram到Transformer的各类模型中表现最优。此外，提出基于轻量级BERT的za_BERT_lid模型，使用NHCLT + Vukzenzele语料训练，性能与最佳非洲特化模型相当。
原文摘要 · Abstract (English)
In this paper, we investigate the use of N-gram models and Large Pre-trained Multilingual models for Language Identification (LID) across 11 South African languages. For N-gram models, this study shows that effective data size selection remains crucial for establishing effective frequency distributions of the target languages, that efficiently model each language, thus, improving language ranking. For pre-trained multilingual models, we conduct extensive experiments covering a diverse set of massively pre-trained multilingual (PLM) models -- mBERT, RemBERT, XLM-r, and Afri-centric multilingual models -- AfriBERTa, Afro-XLMr, AfroLM, and Serengeti. We further compare these models with available large-scale Language Identification tools: Compact Language Detector v3 (CLD V3), AfroLID, GlotLID, and OpenLID to highlight the importance of focused-based LID. From these, we show that Serengeti is a superior model across models: N-grams to Transformers on average. Moreover, we propose a lightweight BERT-based LID model (za_BERT_lid) trained with NHCLT + Vukzenzele corpus, which performs on par with our best-performing Afri-centric models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。