提出零错误边界评估大模型真实计算能力，发现顶级模型连简单计数都常错。
Even GPT-5.2 Can't Count to Five: The Case for Zero-Error Horizons in Trustworthy LLMs
- 用零错误边界衡量模型无错计算最大范围，揭示隐藏缺陷
- GPT-5.2在短字符串奇偶性、括号匹配等基础任务上仍会出错
- 适合关注模型可信性与算法推理能力的研究者
我们提出零错误边界（ZEH）用于评估可信大语言模型，表示模型在不出现任何错误的情况下可解决的最大计算范围。尽管ZEH本身概念简单，但对当前顶级大模型进行评测却揭示了丰富洞见。例如，对GPT-5.2的评测显示，该模型无法正确判断如11000这样的短字符串奇偶性，也无法识别((((())))))中括号是否平衡。这一结果令人意外，因GPT-5.2在其他任务上表现优异。此类在基础逻辑任务上的失误，提醒我们在安全关键领域使用大模型时需保持警惕。通过对Qwen2.5应用ZEH并深入分析，我们发现：虽然ZEH与整体准确率相关，但具体行为模式存在差异，且能为算法能力的涌现提供线索。尽管计算ZEH代价高昂，我们提出通过树结构与在线Softmax实现最高达一个数量级的速度提升。
原文摘要 · Abstract (English)
We propose Zero-Error Horizon (ZEH) for trustworthy LLMs, which represents the maximum range that a model can solve without any errors. While ZEH itself is simple, we demonstrate that evaluating the ZEH of state-of-the-art LLMs yields abundant insights. For example, by evaluating the ZEH of GPT-5.2, we found that GPT-5.2 cannot even compute the parity of a short string like 11000, and GPT-5.2 cannot determine whether the parentheses in ((((()))))) are balanced. This is surprising given the excellent capabilities of GPT-5.2. The fact that LLMs make mistakes on such simple problems serves as an important lesson when applying LLMs to safety-critical domains. By applying ZEH to Qwen2.5 and conducting detailed analysis, we found that while ZEH correlates with accuracy, the detailed behaviors differ, and ZEH provides clues about the emergence of algorithmic capabilities. Finally, while computing ZEH incurs significant computational cost, we discuss how to mitigate this cost by achieving up to one order of magnitude speedup using tree structures and online softmax.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。