研究大模型是否会误把相关当因果,发现不同模型表现差异大。
Are UFOs Driving Innovation? The Illusion of Causality in Large Language Models
- 用新闻标题生成测试模型是否把相关关系错认成因果
- Claude-3.5-Sonnet最不容易产生因果错觉,表现最稳
- 当用户带偏见提问时,模型更易犯错,但该模型仍最抗干扰
认知偏差中的因果错觉指人们在无证据时误认为两变量存在因果关系。本研究考察大语言模型在真实场景中是否产生此类错觉。我们对比GPT-4o-Mini、Claude-3.5-Sonnet和Gemini-1.5-Pro生成的新闻标题,评估其是否将相关关系错误表述为因果。同时引入用户偏见提示,检测模型是否因迎合用户而加剧错觉。结果表明,Claude-3.5-Sonnet在相关转因果夸大方面表现最佳,与人类撰写新闻稿实验结果一致;尽管迎合行为会提升因果错觉概率,尤其影响GPT-4o-Mini,但Claude-3.5-Sonnet仍最具鲁棒性。
原文摘要 · Abstract (English)
Illusions of causality occur when people develop the belief that there is a causal connection between two variables with no supporting evidence. This cognitive bias has been proposed to underlie many societal problems including social prejudice, stereotype formation, misinformation and superstitious thinking. In this research we investigate whether large language models develop the illusion of causality in real-world settings. We evaluated and compared news headlines generated by GPT-4o-Mini, Claude-3.5-Sonnet, and Gemini-1.5-Pro to determine whether the models incorrectly framed correlations as causal relationships. In order to also measure sycophantic behavior, which occurs when a model aligns with a user's beliefs in order to look favorable even if it is not objectively correct, we additionally incorporated the bias into the prompts, observing if this manipulation increases the likelihood of the models exhibiting the illusion of causality. We found that Claude-3.5-Sonnet is the model that presents the lowest degree of causal illusion aligned with experiments on Correlation-to-Causation Exaggeration in human-written press releases. On the other hand, our findings suggest that while mimicry sycophancy increases the likelihood of causal illusions in these models, especially in GPT-4o-Mini, Claude-3.5-Sonnet remains the most robust against this cognitive bias.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。