系统评估20个大模型在30种认知偏差下的表现，揭示其决策缺陷。
A Comprehensive Evaluation of Cognitive Biases in LLMs
- 构建通用测试框架，可大规模生成模型偏差检测用例
- 发现至少1个模型存在全部30种偏差，覆盖率达90%以上
- 开源工具链，助力后续模型公平性研究
我们对20个前沿大语言模型（LLMs）在多种决策场景下进行了大规模评估，涵盖30种认知偏差。贡献包括一个新型通用测试框架，可实现可靠、大规模生成测试用例；一个包含3万条测试样本的基准数据集，用于检测LLMs中的认知偏差；以及对20个模型偏差情况的全面评估。研究证实并扩展了先前关于大模型存在认知偏差的结论，报告了所有30种测试偏差至少在部分模型中被观测到。相关框架代码已公开，以促进未来对大模型偏见的研究：https://github.com/simonmalberg/cognitive-biases-in-llms
原文摘要 · Abstract (English)
We present a large-scale evaluation of 30 cognitive biases in 20 state-of-the-art large language models (LLMs) under various decision-making scenarios. Our contributions include a novel general-purpose test framework for reliable and large-scale generation of tests for LLMs, a benchmark dataset with 30,000 tests for detecting cognitive biases in LLMs, and a comprehensive assessment of the biases found in the 20 evaluated LLMs. Our work confirms and broadens previous findings suggesting the presence of cognitive biases in LLMs by reporting evidence of all 30 tested biases in at least some of the 20 LLMs. We publish our framework code to encourage future research on biases in LLMs: https://github.com/simonmalberg/cognitive-biases-in-llms
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。