用认知理论分析大模型推理错误，发现越强的模型越易犯人类常见逻辑谬误。
Theory-Grounded Evaluation of Human-Like Fallacy Patterns in LLM Reasoning
- 基于理性提问理论生成383个形式化推理题，系统测试模型错误模式。
- 模型越强，错误越接近理论预测的逻辑谬误（相关性ρ=0.360，p=0.0265）。
- 调整前提顺序可减少谬误，适用于研究模型推理机制与人类认知差异。
我们通过检验语言模型的错误是否符合已知的人类逻辑谬误模式，来研究其逻辑推理能力。利用理性提问理论（ETR）及其开源实现PyETR，我们程序化生成了383个形式化定义的推理问题，并评估了38个模型。对每个回答，判断其逻辑正确性，若错误，则进一步判断是否匹配ETR预测的谬误类型。主要发现：(i) 随着模型能力（以Chatbot Arena Elo衡量）提升，其错误中符合ETR预测谬误的比例显著上升（ρ=0.360, p=0.0265），而整体正确率与此无相关性；(ii) 逆转前提顺序能显著降低多个模型的谬误生成，与人类顺序效应一致。方法上，PyETR提供了一个开源管道，可生成无限量、合成式、抗污染的推理测试，支持聚焦于错误构成而非错误率的分析。
原文摘要 · Abstract (English)
We study logical reasoning in language models by asking whether their errors follow established human fallacy patterns. Using the Erotetic Theory of Reasoning (ETR) and its open-source implementation, PyETR, we programmatically generate 383 formally specified reasoning problems and evaluate 38 models. For each response, we judge logical correctness and, when incorrect, whether it matches an ETR-predicted fallacy. Two results stand out: (i) as a capability proxy (Chatbot Arena Elo) increases, a larger share of a model's incorrect answers are ETR-predicted fallacies $(ρ=0.360, p=0.0265)$, while overall correctness on this dataset shows no correlation with capability; (ii) reversing premise order significantly reduces fallacy production for many models, mirroring human order effects. Methodologically, PyETR provides an open-source pipeline for unbounded, synthetic, contamination-resistant reasoning tests linked to a cognitive theory, enabling analyses that focus on error composition rather than error rate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。