为乌尔都语构建了高效专用的BERT模型，效果媲美多语言模型。
DunbaaBERT: From Sacrifice to Semantics
- 从头训练3种词表大小的乌尔都语BERT，使用17GB去重语料。
- 32k词表模型在多个任务中表现最佳，效率优势明显。
- 适合需要低成本高效率的乌尔都语自然语言处理研究者。
大语言模型在众多自然语言处理任务中表现优异，但乌尔都语因资源有限和评估标准分散而相对被忽视。为弥补这一差距，我们提出DunbaaBERT，一种基于乌尔都语罗伯塔基础模型的系列模型，采用32k、52k和96k个词元的字节-分词（Byte-BPE）词表，从零开始在17GB去重乌尔都语语料上训练。我们在涵盖语言可接受性、新闻分类、攻击性语言检测和情感分析的内在与下游任务基准上评估DunbaaBERT，并分析词表大小对性能与效率的影响。各变体在不同基准上均达到与强大多语言基线相当的性能，且始终具备良好的效率权衡。有趣的是，更大的词表并未持续提升下游效果，其中DunbaaBERT$_{\text{32k}}$ 反复展现出最强的整体效率表现。总体而言，结果表明，经过精心设计的乌尔都语专用编码器模型，即使在相对紧凑的模型规模和训练量下，仍能保持高度竞争力。所有模型均以MIT许可证发布。
原文摘要 · Abstract (English)
Large language models have achieved strong performance across many NLP tasks, yet Urdu remains comparatively underexplored due to limited resources and fragmented evaluation settings. To address this gap, we introduce DunbaaBERT, a family of Urdu RoBERTa-base models trained from scratch with Byte-BPE vocabularies of 32k, 52k, and 96k tokens on a deduplicated 17GB Urdu corpus. We evaluate DunbaaBERT across intrinsic and downstream Urdu NLP benchmarks covering linguistic acceptability, news classification, offensive language detection, and sentiment analysis while analyzing vocabulary-size effects on performance and efficiency trade-offs. Across benchmarks, the DunbaaBERT variants achieve competitive performance against strong multilingual baselines while consistently maintaining favorable efficiency trade-offs. Interestingly, larger vocabularies do not consistently improve downstream effectiveness, with DunbaaBERT$_{\text{32k}}$ repeatedly providing the strongest overall efficiency profile. Overall, our results demonstrate that carefully curated Urdu-specific encoder models can remain highly competitive despite comparatively compact model and training scales. All models are released under the MIT license.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。