首个针对阿拉伯语大模型幻觉的细粒度评估框架,覆盖问答与摘要任务。
AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs
- 构建12类细粒度幻觉指标,精准衡量阿拉伯语生成任务中的事实一致性。
- 12个模型测试显示,所有模型中事实性幻觉比忠实性错误更普遍。
- 阿拉伯专有模型Allam表现优于多语言模型,接近推理型模型。
近期关于大语言模型(LLMs)幻觉的研究主要集中于英语。尽管多语言及阿拉伯语专用模型日益增多,但阿拉伯语情境下幻觉评估仍相对匮乏。考虑到阿拉伯语在全球通信与媒体中的广泛使用,这一知识缺口尤为突出。本文首次系统评估了阿拉伯语及多语言LLMs在生成式问答(GQA)和摘要两个关键任务上的幻觉表现。共评估12个模型,包括4个阿拉伯预训练模型、4个多语言模型和4个基于推理的模型。为评估输出的事实一致性和忠实性,我们构建了一个包含12种细粒度幻觉指标的评估框架,涵盖各任务的差异化特征。结果表明,所有模型和任务中,事实性幻觉均显著多于忠实性错误。值得注意的是,阿拉伯预训练模型Allam的幻觉率持续低于多语言模型,且表现与推理型模型相当。代码已开源:https://github.com/aishaalansari57/AraHalluEval。
原文摘要 · Abstract (English)
Recently, extensive research on the hallucination of the large language models (LLMs) has mainly focused on the English language. Despite the growing number of multilingual and Arabic-specific LLMs, evaluating LLMs' hallucination in the Arabic context remains relatively underexplored. The knowledge gap is particularly pressing given Arabic's widespread use across many regions and its importance in global communication and media. This paper presents the first comprehensive hallucination evaluation of Arabic and multilingual LLMs on two critical Arabic natural language generation tasks: generative question answering (GQA) and summarization. This study evaluates a total of 12 LLMs, including 4 Arabic pre-trained models, 4 multilingual models, and 4 reasoning-based models. To assess the factual consistency and faithfulness of LLMs' outputs, we developed a fine-grained hallucination evaluation framework consisting of 12 fine-grained hallucination indicators that represent the varying characteristics of each task. The results reveal that factual hallucinations are more prevalent than faithfulness errors across all models and tasks. Notably, the Arabic pre-trained model Allam consistently demonstrates lower hallucination rates than multilingual models and a comparative performance with reasoning-based models. The code is available at: https://github.com/aishaalansari57/AraHalluEval
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。