构建首个尼日利亚皮钦语情感与讽刺识别基准，填补非洲语言AI评估空白。
Wazobia Eval: A Benchmark for Nigerian Pidgin Emotion Understanding, Sarcasm Detection, and Cultural Reasoning
- 基于人工标注550+样本，设计16类文化特异性情绪分类体系。
- 首次系统评估模型对尼日利亚皮钦语情感、讽刺及文化推理能力。
- 适合关注非洲语言AI、低资源语言理解的研究者使用。
尼日利亚皮钦语是非洲使用最广泛的语言之一，但在语言模型评估中仍严重缺失。现有基准主要聚焦翻译、转录或通用情感分析，未能涵盖文化语境下的语言理解关键维度。本文提出Wazobia Eval，一个用于评估尼日利亚皮钦语情感理解、讽刺检测与文化推理能力的基准。该基准基于手动标注的数据集，包含超过550个示例，并设计了16类情绪分类体系，以捕捉传统情感框架未覆盖的文化特异性情感表达。Wazobia Eval提供标准化评估协议与任务，支持模型在复杂尼日利亚语言理解任务上的性能评测。我们详细介绍了基准设计、标注方法、分类体系构建过程及初步试点评估结果。目标是为尼日利亚语言AI建立基础评估基础设施，并推动未来研究的可复现性。数据集已公开于https://huggingface.co/WAZOBIALABS。
原文摘要 · Abstract (English)
Nigerian Pidgin is one of Africa's most widely spoken languages, yet remains severely underrepresented in language model evaluation. Existing benchmarks primarily focus on translation, transcription, or generic sentiment analysis, leaving critical aspects of culturally grounded language understanding unmeasured. We introduce Wazobia Eval, a benchmark for evaluating Nigerian Pidgin emotion understanding, sarcasm detection, and cultural reasoning. The benchmark is built on a manually annotated dataset containing over 550 examples and a 16-category emotion taxonomy designed to capture culturally specific emotional registers that are not represented in conventional sentiment frameworks. Wazobia Eval provides standardized evaluation protocols and benchmark tasks for assessing model performance on nuanced Nigerian language understanding. We present the benchmark design, annotation methodology, taxonomy development process, and preliminary pilot evaluation results. Our goal is to provide foundational evaluation infrastructure for Nigerian language AI and establish a reproducible benchmark for future research. The dataset is publicly available at https://huggingface.co/WAZOBIALABS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。