用Transformer生成可解释的假新闻核查结果,并自动评估解释质量。
Learning to Generate and Evaluate Fact-checking Explanations with Transformers
- 用Transformer模型生成带理由的假新闻核查结论。
- 生成解释的ROUGE-1达47.77,自动评估与人工判断相关性达MCC 0.7。
- 适合需要提升可信度的AI事实核查系统开发者使用。
在数字平台主导的时代,虚假信息传播带来严峻挑战,亟需能评估信息真实性的解决方案。本研究推动可解释人工智能(XAI)发展,提出基于Transformer的事实核查模型,能够生成人类可理解的解释来支持判断。同时,开发了用于自动评估解释质量的模型,涵盖自相矛盾、幻觉、说服力和整体质量等维度。通过引入以人为中心的评估方法并构建专用数据集,强调人工智能解释需与人类判断对齐。实验表明,最优生成模型在提供高质量证据时,ROUGE-1得分为47.77;最优度量学习模型在客观维度(如自相矛盾和幻觉)上与人工判断具有中等强度相关性,马修斯相关系数(MCC)约为0.7。该工作不仅深化了XAI理论认知,也提升了事实核查系统的透明性、可靠性和用户信任,为减少人工评估依赖提供了初步路径。
原文摘要 · Abstract (English)
In an era increasingly dominated by digital platforms, the spread of misinformation poses a significant challenge, highlighting the need for solutions capable of assessing information veracity. Our research contributes to the field of Explainable Artificial Antelligence (XAI) by developing transformer-based fact-checking models that contextualise and justify their decisions by generating human-accessible explanations. Importantly, we also develop models for automatic evaluation of explanations for fact-checking verdicts across different dimensions such as \texttt{(self)-contradiction}, \texttt{hallucination}, \texttt{convincingness} and \texttt{overall quality}. By introducing human-centred evaluation methods and developing specialised datasets, we emphasise the need for aligning Artificial Intelligence (AI)-generated explanations with human judgements. This approach not only advances theoretical knowledge in XAI but also holds practical implications by enhancing the transparency, reliability and users' trust in AI-driven fact-checking systems. Furthermore, the development of our metric learning models is a first step towards potentially increasing efficiency and reducing reliance on extensive manual assessment. Based on experimental results, our best performing generative model \textsc{ROUGE-1} score of 47.77, demonstrating superior performance in generating fact-checking explanations, particularly when provided with high-quality evidence. Additionally, the best performing metric learning model showed a moderately strong correlation with human judgements on objective dimensions such as \texttt{(self)-contradiction and \texttt{hallucination}, achieving a Matthews Correlation Coefficient (MCC) of around 0.7.}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。