构建12.5亿词的普什图语语料库，推动低资源语言NLP发展
PashtoCorp: A 1.25-Billion-Word Corpus, Evaluation Suite, and Reproducible Pipeline for Low-Resource Language Development
- 整合39个来源，构建可复现的普什图语大规模语料库
- 预训练模型在命名实体识别上提升10%相对准确率，覆盖97.9%词汇
- 首次提供普什图语大模型基准，适合低资源语言研究者使用
我们提出PashtoCorp，一个包含12.5亿词的普什图语语料库，该语言有6000万使用者但在自然语言处理中严重缺失。语料库来自39个来源，涵盖7个HuggingFace数据集和32个定制网页爬虫，通过可复现的流水线处理，包括阿拉伯字母分词、SHA-256去重和质量过滤。总规模达12.5亿词，分布在281万篇文档中，比OSCAR中的普什图子集大40倍，比此前最大专用语料库大83倍。在PashtoCorp上继续预训练XLM-R-base，使保留困惑度降低25.1%（从8.08降至6.06）。在WikiANN普什图语命名实体识别任务中，预训练模型相对提升10%（19.0%→21.0%），训练方差降低近7倍；在50句训练样本下提升达27%，且覆盖97.9%的实体词汇。在Belebele普什图语阅读理解任务中，Gemma-3n达到64.6%准确率，为该基准首个公开的大型语言模型基线。留一法消融实验表明，维基百科（占文档0.7%）是命名实体识别最关键来源：仅移除它即导致实体F1下降47%。语料、模型与代码已开源。
原文摘要 · Abstract (English)
We present PashtoCorp, a 1.25-billion-word corpus for Pashto, a language spoken by 60 million people that remains severely underrepresented in NLP. The corpus is assembled from 39 sources spanning seven HuggingFace datasets and 32 purpose-built web scrapers, processed through a reproducible pipeline with Arabic-script tokenization, SHA-256 deduplication, and quality filtering. At 1.25B words across 2.81 million documents, PashtoCorp is 40x larger than the OSCAR Pashto subset and 83x larger than the previously largest dedicated Pashto corpus. Continued MLM pretraining of XLM-R-base on PashtoCorp reduces held-out perplexity by 25.1% (8.08->6.06). On WikiANN Pashto NER, the pretrained model improves entity F1 by 10% relative (19.0%->21.0%) and reduces training variance nearly 7x; the largest gain appears at 50 training sentences (+27%), with PashtoCorp covering 97.9% of WikiANN entity vocabulary. On Belebele Pashto reading comprehension, Gemma-3n achieves 64.6% accuracy, the first published LLM baseline for Pashto on this benchmark. A leave-one-out source ablation shows that Wikipedia (0.7% of documents) is the most critical source for NER: removing it alone reduces entity F1 by 47%. Corpus data, trained model, and code are available at https://huggingface.co/datasets/ihanif/pashto-corpus, https://huggingface.co/ihanif/xlmr-pashto, and https://github.com/ihanif/pashto-corpus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。