为资源匮乏的绍纳语构建了开源形态分析工具,提升其自然语言处理能力。
Shona spaCy: A Morphological Analyzer for an Under-Resourced Bantu Language
- 基于规则和词典的形态分析框架,整合名词类前缀、动词主语一致等语言特征。
- 在正式与非正式语料上实现90%词性标注准确率和88%形态特征准确率。
- 透明可解释的系统设计,适合研究绍纳语及类似非洲语言的学者与开发者。
尽管多语言自然语言处理快速进展,绍纳语在形态分析和语言感知工具方面仍严重不足。本文提出Shona spaCy,一个基于spaCy框架的开源、基于规则的绍纳语形态分析工具。该系统结合精心整理的JSON词典与语言学驱动规则,建模名词类前缀(Mupanda 1-18)、动词主语一致、时态体标记、拟声词和附着成分,并将这些信息整合为词元级别的词根、词性及形态特征标注。工具可通过pip install shona-spacy安装,源代码位于https://github.com/HappymoreMasoka/shona-spacy,PyPI发布版为https://pypi.org/project/shona-spacy/0.1.4/。在正式与非正式绍纳语文本语料上的评估显示,词性标注准确率达90%,形态特征准确率为88%,同时保持语言决策的透明性。通过连接描述语法与计算实现,Shona spaCy推动了绍纳语的自然语言处理普及与数字包容性,为其他资源匮乏的班图语言提供形态分析工具模板。
原文摘要 · Abstract (English)
Despite rapid advances in multilingual natural language processing (NLP), the Bantu language Shona remains under-served in terms of morphological analysis and language-aware tools. This paper presents Shona spaCy, an open-source, rule-based morphological pipeline for Shona built on the spaCy framework. The system combines a curated JSON lexicon with linguistically grounded rules to model noun-class prefixes (Mupanda 1-18), verbal subject concords, tense-aspect markers, ideophones, and clitics, integrating these into token-level annotations for lemma, part-of-speech, and morphological features. The toolkit is available via pip install shona-spacy, with source code at https://github.com/HappymoreMasoka/shona-spacy and a PyPI release at https://pypi.org/project/shona-spacy/0.1.4/. Evaluation on formal and informal Shona corpora yields 90% POS-tagging accuracy and 88% morphological-feature accuracy, while maintaining transparency in its linguistic decisions. By bridging descriptive grammar and computational implementation, Shona spaCy advances NLP accessibility and digital inclusion for Shona speakers and provides a template for morphological analysis tools for other under-resourced Bantu languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。