测试大模型在道德条件推理中的表现,发现其错误模式像人类。
Evaluation of Deontic Conditional Reasoning in Large Language Models: The Case of Wason's Selection Task
- 用带道德模态的沃森选择任务数据集评估大模型
- 大模型在道德规则下表现更好,但仍有匹配偏差错误
- 适合研究认知偏见或大模型推理机制的学者
随着大型语言模型(LLMs)语言能力提升,其推理能力日益受到关注。人类在特定领域推理中表现良好,尤其是在规范性而非纯粹形式性的语境中。尽管已有研究对比了大模型与人类的推理能力,但大模型推理的领域特异性仍缺乏系统探索。本研究引入一个新的沃森选择任务数据集,明确编码道德模态以系统区分道德条件句与描述性条件句,并用于检验大模型在道德规则下的条件推理表现。进一步分析显示,观察到的错误模式更符合匹配偏差(忽略否定,选择与规则字面匹配的项目),而非确认偏差。结果表明,大模型的表现随规则类型系统性变化,且其错误模式可类比于人类在该范式中已知的认知偏见。
原文摘要 · Abstract (English)
As large language models (LLMs) advance in linguistic competence, their reasoning abilities are gaining increasing attention. In humans, reasoning often performs well in domain specific settings, particularly in normative rather than purely formal contexts. Although prior studies have compared LLM and human reasoning, the domain specificity of LLM reasoning remains underexplored. In this study, we introduce a new Wason Selection Task dataset that explicitly encodes deontic modality to systematically distinguish deontic from descriptive conditionals, and use it to examine LLMs' conditional reasoning under deontic rules. We further analyze whether observed error patterns are better explained by confirmation bias (a tendency to seek rule-supporting evidence) or by matching bias (a tendency to ignore negation and select items that lexically match elements of the rule). Results show that, like humans, LLMs reason better with deontic rules and display matching-bias-like errors. Together, these findings suggest that the performance of LLMs varies systematically across rule types and that their error patterns can parallel well-known human biases in this paradigm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。