文本中的谎言难以被准确识别,跨语言实验表明现有方法大多失效。
What if Deception Cannot be Detected? A Cross-Linguistic Study on the Limits of Deception Detection from Text
- 用信念偏差框架分离谎言与事实,避免数据污染干扰
- 三组跨语言数据中语言线索与谎言无显著关联
- 主流模型在新数据上表现接近随机,提示研究需重审
谎言能否仅从文本中识别?现有研究声称能成功自动识别,但我们怀疑其结果受数据收集过程的伪影影响,无法泛化。本文提出基于信念偏差的谎言定义——即作者陈述与其真实信念不一致,不论事实对错,从而可孤立分析欺骗线索。据此构建三个语料库(统称DeFaBel),包括德语和英德双语版本,分别在不同条件下采集以捕捉信念变化,支持跨语言分析。评估常见语言线索后发现,在所有三个DeFaBel变体中,这些线索与谎言标签的相关性极低且统计不显著,与先前研究结论相悖。进一步在遵循相似采集流程的英文数据集上测试,虽部分有显著相关性,但效应量小且预测线索不一致。使用特征模型、预训练语言模型及指令微调大模型进行检测,尽管在已有数据集上表现良好,但在DeFaBel上始终接近随机水平。研究挑战了‘语言线索可可靠推断谎言’的假设,呼吁重新思考自然语言处理中谎言的研究与建模方式。
原文摘要 · Abstract (English)
Can deception be detected solely from written text? Cues of deceptive communication are inherently subtle, even more so in text-only communication. Yet, prior studies have reported considerable success in automatic deception detection. We hypothesize that such findings are largely driven by artifacts introduced during data collection and do not generalize beyond specific datasets. We revisit this assumption by introducing a belief-based deception framework, which defines deception as a misalignment between an author's claims and true beliefs, irrespective of factual accuracy, allowing deception cues to be studied in isolation. Based on this framework, we construct three corpora, collectively referred to as DeFaBel, including a German-language corpus of deceptive and non-deceptive arguments and a multilingual version in German and English, each collected under varying conditions to account for belief change and enable cross-linguistic analysis. Using these corpora, we evaluate commonly reported linguistic cues of deception. Across all three DeFaBel variants, these cues show negligible, statistically insignificant correlations with deception labels, contrary to prior work that treats such cues as reliable indicators. We further benchmark against other English deception datasets following similar data collection protocols. While some show statistically significant correlations, effect sizes remain low and, critically, the set of predictive cues is inconsistent across datasets. We also evaluate deception detection using feature-based models, pretrained language models, and instruction-tuned large language models. While some models perform well on established deception datasets, they consistently perform near chance on DeFaBel. Our findings challenge the assumption that deception can be reliably inferred from linguistic cues and call for rethinking how deception is studied and modeled in NLP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。