构建首个大规模约鲁巴语数据集，助力非洲语言NLP研究
Yankari: A Monolingual Yoruba Dataset
- 从13个来源收集5.1万篇文档，超3000万词元
- 采用自动化质量控制与严格清洗，确保数据可靠性
- 为约鲁巴语模型开发与数字平等提供基础资源
本文介绍Yankari，一个大规模单语约鲁巴语数据集，旨在填补该重要西非语言在自然语言处理（NLP）资源上的关键空白。尽管约鲁巴语有超过3000万使用者，但在NLP研究与应用中仍严重缺失。我们详细阐述了数据集的构建方法，包括精心筛选来源、自动化质量控制及严格的清理流程。Yankari包含来自13个不同来源的51,407份文档，总计超过3000万词元。我们的方法注重伦理采集，避免问题来源并解决现有数据集的常见问题。我们提供了全面的自动化评估，证明其相对于现有资源的质量优势。Yankari代表了约鲁巴语资源的重大进展，为开发更精准的NLP模型、支持对比语言学研究以及推动约鲁巴语数字可及性奠定了基础。
原文摘要 · Abstract (English)
This paper presents Yankari, a large-scale monolingual dataset for the Yoruba language, aimed at addressing the critical gap in Natural Language Processing (NLP) resources for this important West African language. Despite being spoken by over 30 million people, Yoruba has been severely underrepresented in NLP research and applications. We detail our methodology for creating this dataset, which includes careful source selection, automated quality control, and rigorous data cleaning processes. The Yankari dataset comprises 51,407 documents from 13 diverse sources, totaling over 30 million tokens. Our approach focuses on ethical data collection practices, avoiding problematic sources and addressing issues prevalent in existing datasets. We provide thorough automated evaluations of the dataset, demonstrating its quality compared to existing resources. The Yankari dataset represents a significant advancement in Yoruba language resources, providing a foundation for developing more accurate NLP models, supporting comparative linguistic studies, and contributing to the digital accessibility of the Yoruba language.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。