首个多语言幻觉评估基准，揭示小模型在低资源语言中幻觉率超60%。
A multilingual hallucination benchmark: MultiWikiQHalluA
- 基于多语言数据集构建306种语言的合成幻觉数据，训练欧洲30国语言的幻觉检测模型。
- 小模型Qwen3-0.6B在冰岛语中幻觉率高达60%，大模型表现更优，尤其70B级模型最佳。
- 发现低资源语言幻觉问题更严重，为跨语言模型可靠性提供关键实证。
现有幻觉评估多集中于英语，难以判断结论是否适用于低资源语言。本文研究忠实性幻觉——即模型生成内容流畅且合理，但与输入不符或内部不一致。利用多语言MultiWikiQA数据集，通过LettuceDetect框架生成306种语言的合成幻觉数据，训练出30种欧洲语言的词级别幻觉分类器。评估了Qwen3-0.6B、Qwen3-14B、Gemma-3-12B-IT、cogito-v1-preview-qwen-32B和cogito-v1-preview-llama-70B在英语、丹麦语、德语和冰岛语上的幻觉率。结果显示，Qwen3-0.6B在冰岛语中幻觉率最高达60%（至少一个幻觉的答案占比），大模型整体表现更好，其中cogito-v1-preview-qwen-32B和cogito-v1-preview-llama-70B在多数语言上最优。低资源语言幻觉率显著更高，尤以冰岛语为甚。
原文摘要 · Abstract (English)
Most hallucination evaluations focus on English, leaving it unclear whether findings transfer to lower-resource languages. We investigate faithfulness hallucinations, defined as model-generated content that is fluent and plausible but diverges from the provided input or is internally inconsistent. Leveraging the multilingual MultiWikiQA dataset, we utilize the LettuceDetect framework to create synthetic hallucination datasets for 306 languages, from which we train token-level hallucination classifiers for 30 European languages. In this work, we present evaluations of model hallucinations on a selection of languages: English, Danish, German, and Icelandic. Using these classifiers, we evaluate the hallucination rates for Qwen3-0.6B, Qwen3-14B, Gemma-3-12B-IT, cogito-v1-preview-qwen-32B, and cogito-v1-preview-llama-70B. Our classifiers reveal notably higher hallucination rates for Qwen3-0.6B (up to 60\% of answers containing at least one hallucination, peaking in Icelandic) and generally lower rates for larger models, with cogito-v1-preview-qwen-32B and cogito-v1-preview-llama-70B performing best on most languages. Hallucination rates are consistently higher for lower-resource languages, particularly Icelandic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。