测试大模型执行复杂步骤的能力，发现越长的步骤越容易出错。
When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models
- 设计可控诊断任务，让模型按步骤计算数值
- 95步任务准确率仅20%，远低于5步时的63%
- 常见错误包括漏算、提前输出、自纠正和执行不全
大型语言模型在推理基准上表现良好，但最终答案准确率无法反映其是否忠实执行提示中的步骤。我们引入一个受控的诊断基准，用于评估算术过程执行能力：模型需根据分步算术流程和两个数值输入，返回最终结果。通过流程长度和中间变量回溯依赖关系调节复杂度。平均首次回答准确率从5步任务的63%下降至95步任务的20%。生成层面分析显示，失败多表现为缺失答案、过早输出、初始错误后的自我修正以及执行不完整。这些发现揭示了随着算术过程复杂度增加，执行性能持续下降的现象。
原文摘要 · Abstract (English)
Large language models (LLMs) often achieve strong performance on reasoning benchmarks, but final-answer accuracy alone does not show whether they faithfully execute the procedure specified in a prompt. We introduce a controlled diagnostic benchmark for arithmetic procedural execution, where models are given a step-wise arithmetic procedure and two numeric inputs, and must return the final computed value. Complexity is varied through procedure length and look-back dependencies over intermediate variables. Average first-answer accuracy drops from 63% on 5-step procedures to 20\% on 95-step procedures. Generation-level analysis shows that failures often involve missing answers, premature answers, self-correction after an initial error and under-executed traces. These findings reveal a consistent decline in execution performance as arithmetic procedural complexity increases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。