区分事实性与欺骗性,发现德语文本中两者无相关性
How Entangled is Factuality and Deception in German?
- 用信念框架分离陈述事实与欺骗行为,避免混淆
- 模型在德语欺骗检测上表现如随机猜测,远低于预期
- 大模型提示法对非事实和欺骗内容更鲁棒,适合实际应用
‘地球是平的’这一说法在事实上错误,但如果某人真心相信并为之辩护,则不构成欺骗。现有欺骗检测与事实核查研究常将事实准确性与陈述真实性混为一谈,导致难以分析二者间的细微差异及其对下游任务的影响。本文采用基于信念的欺骗框架,将文本定义为欺骗仅当其言辞与其真实信念不符。研究评估该框架在德语文本中的适用性,测试计算模型在已有信念型论据语料上的欺骗检测效果,并考察欺骗对事实核查任务的影响。结果出乎意料:既有的欺骗线索在德语文本中无显著相关性。先前研究声称模型可超越人类,但实验表明传统及最先进的模型在该任务上表现不佳,接近随机水平。在事实核查方面,基于自然语言推理的验证模型在非事实与欺骗内容上表现更差;而通过提示引导的大语言模型则对此类属性更具鲁棒性。
原文摘要 · Abstract (English)
The statement "The earth is flat" is factually inaccurate, but if someone truly believes and argues in its favor, it is not deceptive. Research on deception detection and fact checking often conflates factual accuracy with the truthfulness of statements. This assumption makes it difficult to (a) study subtle distinctions and interactions between the two and (b) gauge their effects on downstream tasks. The belief-based deception framework disentangles these properties by defining texts as deceptive when there is a mismatch between what people say and what they truly believe. In this study, we assess if presumed patterns of deception generalize to German language texts. We test the effectiveness of computational models in detecting deception using an established corpus of belief-based argumentation. Finally, we gauge the impact of deception on the downstream task of fact checking and explore if this property confounds verification models. Surprisingly, our analysis finds no correlation with established cues of deception. Previous work claimed that computational models can outperform humans in deception detection accuracy, however, our experiments show that both traditional and state-of-the-art models struggle with the task, performing no better than random guessing. For fact checking, we find that Natural Language Inference-based verification performs worse on non-factual and deceptive content, while prompting Large Language Models for the same task is less sensitive to these properties.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。