arXiv:2509.22856cs.CL2025-09被引 8

评估45个大模型在8类认知偏见上的表现,发现模型普遍存在系统性偏差。

The Bias is in the Details: An Assessment of Cognitive Bias in LLMs

  • 设计多选题框架与心理学合作构建220个决策场景
  • 17.8%-57.3%的响应呈现偏见行为,部分受模型规模和提示细节影响
  • 大模型可降低偏见,但特定提示会加剧某些偏见,适合可信度研究者参考

随着大语言模型(LLMs)越来越多地嵌入现实决策过程,考察其是否表现出认知偏见变得至关重要。本文对45个大语言模型在八种经典认知偏见上进行大规模评估,通过控制提示变化生成超过280万条响应。研究引入基于多选题的评估框架,与心理学家合作构建包含220个决策场景的数据集,并提出从人类编写模板中生成多样化提示的可扩展方法。分析显示,在锚定、可得性、证实、框架、解释、过度归因、前景理论和代表性等偏见情境中,模型在17.8%至57.3%的实例中表现出一致于偏见的行为。结果表明,模型规模(>32B参数)可在39.5%情况下减少偏见,而更高提示细节可将多数偏见降低最多14.9%,但在过度归因情形下反而加剧达8.8%。

原文摘要 · Abstract (English)

As Large Language Models (LLMs) are increasingly embedded in real-world decision-making processes, it becomes crucial to examine the extent to which they exhibit cognitive biases. Extensively studied in the field of psychology, cognitive biases appear as systematic distortions commonly observed in human judgments. This paper presents a large-scale evaluation of eight well-established cognitive biases across 45 LLMs, analyzing over 2.8 million LLM responses generated through controlled prompt variations. To achieve this, we introduce a novel evaluation framework based on multiple-choice tasks, hand-curate a dataset of 220 decision scenarios targeting fundamental cognitive biases in collaboration with psychologists, and propose a scalable approach for generating diverse prompts from human-authored scenario templates. Our analysis shows that LLMs exhibit bias-consistent behavior in 17.8-57.3% of instances across a range of judgment and decision-making contexts targeting anchoring, availability, confirmation, framing, interpretation, overattribution, prospect theory, and representativeness biases. We find that both model size and prompt specificity play a significant role on bias susceptibility as follows: larger size (>32B parameters) can reduce bias in 39.5% of cases, while higher prompt detail reduces most biases by up to 14.9%, except in one case (Overattribution), which is exacerbated by up to 8.8%.

认知偏见大模型评估心理机制提示工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。