构建首个马拉地语情绪识别数据集，用大模型合成标注提升低资源语言研究。
L3Cube-MahaEmotions: A Marathi Emotion Recognition Dataset with Synthetic Annotations using CoTR prompting and Large Language Models
- 用CoTR提示技术结合GPT-4生成马拉地语情绪标注数据
- 11类细粒度情绪标签，验证与测试集由人工标注为黄金标准
- 发现通用大模型比微调BERT更适用于复杂情绪识别任务
马拉地语等低资源语言的情绪识别因标注数据有限而困难。本文提出L3Cube-MahaEmotions，一个包含11种细粒度情绪标签的高质量马拉地语情绪识别数据集。训练数据通过大语言模型（LLM）合成标注，验证与测试集则由人工标注，构成可靠的黄金标准基准。基于MahaSent数据集，采用链式翻译（CoTR）提示策略：将马拉地语句子翻译为英文，再通过单个提示进行情绪标注。评估了GPT-4和Llama3-405B，最终选用GPT-4生成训练数据，因其标注质量更优。使用标准指标评估模型性能，并探索了标签聚合策略（如并集、交集）。结果显示，GPT-4预测优于微调的BERT模型，但基于合成标签训练的BERT模型仍无法超越GPT-4。这凸显高质量人工标注数据的重要性及情绪识别的内在复杂性。重要发现是，GPT-4和Llama3-405B等通用大模型在复杂低资源情绪识别任务中表现优于微调的BERT。数据集与模型已公开于https://github.com/l3cube-pune/MarathiNLP。
原文摘要 · Abstract (English)
Emotion recognition in low-resource languages like Marathi remains challenging due to limited annotated data. We present L3Cube-MahaEmotions, a high-quality Marathi emotion recognition dataset with 11 fine-grained emotion labels. The training data is synthetically annotated using large language models (LLMs), while the validation and test sets are manually labeled to serve as a reliable gold-standard benchmark. Building on the MahaSent dataset, we apply the Chain-of-Translation (CoTR) prompting technique, where Marathi sentences are translated into English and emotion labeled via a single prompt. GPT-4 and Llama3-405B were evaluated, with GPT-4 selected for training data annotation due to superior label quality. We evaluate model performance using standard metrics and explore label aggregation strategies (e.g., Union, Intersection). While GPT-4 predictions outperform fine-tuned BERT models, BERT-based models trained on synthetic labels fail to surpass GPT-4. This highlights both the importance of high-quality human-labeled data and the inherent complexity of emotion recognition. An important finding of this work is that generic LLMs like GPT-4 and Llama3-405B generalize better than fine-tuned BERT for complex low-resource emotion recognition tasks. The dataset and model are shared publicly at https://github.com/l3cube-pune/MarathiNLP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。