arXiv:2509.14249cs.CLcs.AI2025-09

构建绍纳语俚语数据集,提升非洲语言对话AI的包容性。

Advancing Conversational AI with Shona Slang: A Dataset and Hybrid Model for Digital Inclusion

  • 收集社交媒体中的绍纳语-英语混用俚语,标注意图、情绪等五类信息
  • 用多语言DistilBERT模型实现96.4%准确率的意图识别
  • 融合规则与检索生成的混合聊天机器人,更适合学生咨询场景

非洲语言在自然语言处理中仍严重缺乏代表性,现有语料多局限于正式语体,无法反映日常交流的鲜活特征。本文针对津巴布韦和赞比亚使用的班图语绍纳语,从匿名社交媒体对话中构建了首个绍纳语-英语俚语数据集,涵盖意图、情感、对话行为、代码混用和语气等标注,已公开于https://github.com/HappymoreMasoka/Working_with_shona-slang。我们基于多语言DistilBERT微调了意图识别分类器,在测试集上达到96.4%准确率与96.3% F1值,模型托管于https://huggingface.co/HappymoreMasoka。该分类器被集成进一个混合聊天机器人,结合规则响应与检索增强生成(RAG),用于支持佩斯大学研究生项目咨询。定性评估显示,该系统在文化相关性和用户参与度上优于纯RAG基线。通过发布数据集、模型与方法,本工作推动了非洲语言NLP资源的发展,促进更具包容性与文化契合度的对话AI。

原文摘要 · Abstract (English)

African languages remain underrepresented in natural language processing (NLP), with most corpora limited to formal registers that fail to capture the vibrancy of everyday communication. This work addresses this gap for Shona, a Bantu language spoken in Zimbabwe and Zambia, by introducing a novel Shona--English slang dataset curated from anonymized social media conversations. The dataset is annotated for intent, sentiment, dialogue acts, code-mixing, and tone, and is publicly available at https://github.com/HappymoreMasoka/Working_with_shona-slang. We fine-tuned a multilingual DistilBERT classifier for intent recognition, achieving 96.4\% accuracy and 96.3\% F1-score, hosted at https://huggingface.co/HappymoreMasoka. This classifier is integrated into a hybrid chatbot that combines rule-based responses with retrieval-augmented generation (RAG) to handle domain-specific queries, demonstrated through a use case assisting prospective students with graduate program information at Pace University. Qualitative evaluation shows the hybrid system outperforms a RAG-only baseline in cultural relevance and user engagement. By releasing the dataset, model, and methodology, this work advances NLP resources for African languages, promoting inclusive and culturally resonant conversational AI.

对话AI非洲语言混合模型数据集

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。