评估编码大模型对软件运行资源的理解能力
Towards Evaluation of Implicit Software World Models in Coding LLMs
- 从控制流扩展到内存、时间等执行资源预测
- 模型在真实任务上表现差且不稳定
- 适合关注AI编程可信度的研究者
软件工程无论是由人类还是AI代理完成，都需要对软件行为进行推理。我们称支持这种推理的内部模型为软件世界模型，并将现有的代码执行基准视为其中一个重要方面——控制流。本文迈出更全面评估的第一步，将可观测维度转向执行资源：除了测试结果和异常类型外，还预测方法级和行级的峰值内存、运行时长及排序后的性能分析输出。数据来自SWE-bench Verified，以确保测试贴近真实软件工程任务。所有测试模型，包括前沿模型，均表现出中等性能和脆弱行为，表明它们对软件实际执行过程的理解明显不足，远未达到对源码写法的理解水平。
原文摘要 · Abstract (English)
Software engineering, whether performed by humans or by AI agents, requires reasoning about how software behaves. We call the internal model that supports such reasoning the software world model, and view current code-execution benchmarks as covering one well-studied slice of it -- control flow. In this paper, we take a step toward a broader evaluation by shifting the observable axis to execution resources: alongside test outcome and exception class, we predict peak memory, wall-clock time, and ranked profiler outputs at method and line granularity. We use SWE-bench Verified as the source of data to hold the test close to real-world software engineering tasks. All tested models, frontier ones included, show modest performance and brittle behaviour, suggesting a notable lack of understanding of how software is executed, as opposed to how its source code is written.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。