arXiv:2509.09677cs.AI2025-09被引 57

大模型执行长任务能力随规模提升,而非逐渐失效。

The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs

  • 通过提供计划与知识,分离执行能力进行评估
  • 大模型在多步任务中显著优于小模型,即使单步准确率相近
  • 思考机制可缓解自我强化错误,支持更长任务执行

大型语言模型(LLMs)持续扩展是否带来收益递减?本文指出,短期任务基准可能产生进展放缓的错觉:即便单步准确率微增,也能在长任务中累积为指数级性能提升。我们发现,当简单任务变长时,模型失败主要源于执行错误而非推理能力不足。为此,我们提出通过显式提供知识和计划来隔离执行能力。实验显示,更大模型在多轮任务中正确执行的回合数显著更多,即使小模型单步准确率已接近完美。同时观察到,随着步骤增加,模型每步准确率下降,这不仅是长上下文限制所致,更存在自条件效应——若上下文包含自身历史错误,模型更易犯错,且此现象不随模型规模扩大而缓解。但引入思考机制可有效减轻该效应,并实现单次调用完成超长任务执行。最终,我们在前沿思考模型上测试了其单次调用能执行的任务长度。研究强调,聚焦执行能力有助于解释为何大模型能在复杂推理中表现优异却在延长任务中失败,并凸显模型规模与序列测试时计算量对长时程任务的巨大价值。

原文摘要 · Abstract (English)

Does continued scaling of large language models (LLMs) yield diminishing returns? In this work, we show that short-task benchmarks may give an illusion of slowing progress, as even marginal gains in single-step accuracy can compound into exponential improvements in the length of tasks a model can successfully complete. Then, we argue that failures of LLMs when simple tasks are made longer arise from mistakes in execution, rather than an inability to reason. So, we propose isolating execution capability, by explicitly providing the knowledge and plan needed to solve a long-horizon task. First, we find that larger models can correctly execute significantly more turns even when small models have near-perfect single-turn accuracy. We then observe that the per-step accuracy of models degrades as the number of steps increases. This is not just due to long-context limitations -- curiously, we observe a self-conditioning effect -- models become more likely to make mistakes when the context contains their errors from prior turns. Self-conditioning does not reduce by just scaling the model size. But, we find that thinking mitigates self-conditioning, and also enables execution of much longer tasks in a single turn. We conclude by benchmarking frontier thinking models on the length of tasks they can execute in a single turn. Overall, by focusing on the ability to execute, we hope to reconcile debates on how LLMs can solve complex reasoning problems yet fail at simple tasks when made longer, and highlight the massive benefits of scaling model size and sequential test-time compute for long-horizon tasks.

大模型执行能力长任务思考机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。