揭示推理模型评估中的实验陷阱，指出误判源于设计缺陷而非能力不足。
Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- 改用生成函数替代穷举步骤，避免输出超限问题。
- 发现原实验中部分题目根本无解，模型失败属误判。
- 强调评估需排除技术限制，才能真实反映推理能力。
Shojaee 等人（2025）报告大型推理模型（LRMs）在超过一定复杂度的规划谜题上出现“准确率坍塌”。我们证明其发现主要源于实验设计缺陷，而非模型根本性推理失败。分析揭示三大问题：(1) 汉诺塔实验易超出模型输出令牌限制，模型自身在输出中明确提及此约束；(2) 原作者的自动化评估框架无法区分推理失败与实际约束，导致能力误判；(3) 最严重的是，其河渡难题基准对 N > 5 的情形包含数学上不可解实例（因船容量不足），但模型被判定为失败。通过控制这些实验伪影，改用生成函数而非完整移动序列请求，初步实验显示多个模型在原报告完全失败的汉诺塔实例上仍保持高准确率。结果强调，评估 AI 推理能力时必须谨慎设计实验。
原文摘要 · Abstract (English)
Shojaee et al. (2025) report that Large Reasoning Models (LRMs) exhibit "accuracy collapse" on planning puzzles beyond certain complexity thresholds. We demonstrate that their findings primarily reflect experimental design limitations rather than fundamental reasoning failures. Our analysis reveals three critical issues: (1) Tower of Hanoi experiments risk exceeding model output token limits, with models explicitly acknowledging these constraints in their outputs; (2) The authors' automated evaluation framework fails to distinguish between reasoning failures and practical constraints, leading to misclassification of model capabilities; (3) Most concerningly, their River Crossing benchmarks include mathematically impossible instances for N > 5 due to insufficient boat capacity, yet models are scored as failures for not solving these unsolvable problems. When we control for these experimental artifacts, by requesting generating functions instead of exhaustive move lists, preliminary experiments across multiple models indicate high accuracy on Tower of Hanoi instances previously reported as complete failures. These findings highlight the importance of careful experimental design when evaluating AI reasoning capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。