用大模型预测代码执行结果，探索其作为通用代理的潜力
SURGE: On the Potential of Large Language Models as General-Purpose Surrogate Code Executors
- 构建涵盖8类任务的1160个问题基准测试
- 21个大模型在复杂代码任务中表现超越传统方法
- 适合对代码智能与模型泛化能力感兴趣的开发者
神经代理模型在数据挖掘中具有强大且高效的优势。与此同时，大语言模型（LLMs）在代码生成与理解等任务中展现出卓越能力。然而，一个同样重要却未被充分探索的问题是：LLMs能否作为代码执行预测的代理模型？为系统性地研究该问题，我们提出了SURGE——一个包含1160个问题的综合性基准，覆盖八大关键领域：多语言编程任务、竞赛级编程题、仓库级代码分析、高成本科学计算、时间复杂度密集型算法、含错代码分析、依赖特定编译器或运行环境的程序，以及形式化数学证明验证。通过对21个开源与专有大模型的广泛分析，我们考察了缩放规律、数据效率和预测准确率。研究揭示了将大模型作为计算过程高效代理的重要可行性。基准与评估框架已公开于https://github.com/Imbernoulli/SURGE。
原文摘要 · Abstract (English)
Neural surrogate models are powerful and efficient tools in data mining. Meanwhile, large language models (LLMs) have demonstrated remarkable capabilities in code-related tasks, such as generation and understanding. However, an equally important yet underexplored question is whether LLMs can serve as surrogate models for code execution prediction. To systematically investigate it, we introduce SURGE, a comprehensive benchmark with $1160$ problems covering $8$ key aspects: multi-language programming tasks, competition-level programming problems, repository-level code analysis, high-cost scientific computing, time-complexity-intensive algorithms, buggy code analysis, programs dependent on specific compilers or execution environments, and formal mathematical proof verification. Through extensive analysis of $21$ open-source and proprietary LLMs, we examine scaling laws, data efficiency, and predictive accuracy. Our findings reveal important insights about the feasibility of LLMs as efficient surrogates for computational processes. The benchmark and evaluation framework are available at https://github.com/Imbernoulli/SURGE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。