检测阿拉伯语伊斯兰问答中的幻觉并找回真实答案
Detecting Hallucinations and Recovering Verified Answers in Arabic Islamic Question Answering
- 基于微调的Gemma模型，采用确定性解码判断回答是否虚构
- 幻觉检测宏平均F1达0.928，答案选择准确率0.895
- 适合关注宗教问答可信度与大模型幻觉治理的研究者
大型语言模型在回答伊斯兰问题时虽能生成流畅文本，但常引入难以识别的事实错误。本文针对 extsc{HalluScoring 2026}任务2.1（伊斯兰幻觉检测与寻真）提出系统方案。该任务要求统一进行两步预测：判断阿拉伯语回答是否为幻觉，并从六个相关候选答案中选出真实答案。我们使用共享任务提供的伊斯兰知识数据集，包含600个问答实例，其中341个为幻觉回答，259个为非幻觉回答。系统基于微调的 exttt{google/gemma-4-12B-it}模型，推理时采用确定性解码。生成结果经规范化处理后提取幻觉标签与选项选择。系统在幻觉检测上取得0.928的宏平均F1和0.935的标签准确率，在答案选择上达到0.895的准确率，综合得分为0.912，表现优异。答案选择准确率低于幻觉检测，说明从合理选项中区分真实答案更具挑战性。
原文摘要 · Abstract (English)
Large language models can generate fluent responses to Islamic questions while introducing factual errors that are difficult to identify. This paper presents our system for \textsc{HalluScoring 2026} Task 2.1, \textit{Islamic Hallucination Detection and Find the Truth}. The task requires a unified two-step prediction: determining whether an Arabic answer generated by an LLM is hallucinated and selecting the verified answer from six closely related candidate options. We use the Islamic knowledge dataset provided by the shared task, which contains 600 question--answer instances, including 341 hallucinated and 259 non-hallucinated answers. Our system is based on the fine-tuned \texttt{google/gemma-4-12B-it} model and uses deterministic decoding during inference. The generated outputs are normalized to extract the hallucination label and the selected option. The system achieves a Macro-F1 score of 0.928 and a label accuracy of 0.935 for hallucination detection, together with an option accuracy of 0.895 for answer selection. These results yield a combined score of 0.912, demonstrating strong performance across both stages of the task. The lower option-selection accuracy indicates that distinguishing the verified answer from plausible alternatives remains more challenging than detecting hallucinated responses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。