让大模型直接执行代码，突破传统生成局限。
Large Language Models as Code Executors: An Exploratory Study
- 将代码片段直接输入大模型，由其自主执行并返回结果。
- o1模型代码执行准确率超90%，其他模型表现较低。
- 提出逐行指令提示法，显著提升弱模型性能。
大型语言模型（LLM）的能力已从自然语言处理拓展至代码理解与生成等复杂任务。本文首次探索将LLM作为代码执行器的可行性，即直接向模型输入代码片段，由其执行并返回输出结果。我们系统评估了OpenAI的o1、GPT-4o、GPT-3.5、DeepSeek及Qwen-Coder等多种模型在该任务上的表现。结果显示，o1模型在代码执行中准确率超过90%，而其他模型表现相对较低。此外，我们提出迭代指令提示（IIP）技术，通过逐行处理代码，平均提升弱模型准确率7.22%（最高达18.96%），相较思维链提示（CoT）绝对提升3.86%（最高达19.46%）。本研究不仅揭示了大模型在编程任务中的变革潜力，也为自动化编程和复杂任务完成奠定了基础。
原文摘要 · Abstract (English)
The capabilities of Large Language Models (LLMs) have significantly evolved, extending from natural language processing to complex tasks like code understanding and generation. We expand the scope of LLMs' capabilities to a broader context, using LLMs to execute code snippets to obtain the output. This paper pioneers the exploration of LLMs as code executors, where code snippets are directly fed to the models for execution, and outputs are returned. We are the first to comprehensively examine this feasibility across various LLMs, including OpenAI's o1, GPT-4o, GPT-3.5, DeepSeek, and Qwen-Coder. Notably, the o1 model achieved over 90% accuracy in code execution, while others demonstrated lower accuracy levels. Furthermore, we introduce an Iterative Instruction Prompting (IIP) technique that processes code snippets line by line, enhancing the accuracy of weaker models by an average of 7.22% (with the highest improvement of 18.96%) and an absolute average improvement of 3.86% against CoT prompting (with the highest improvement of 19.46%). Our study not only highlights the transformative potential of LLMs in coding but also lays the groundwork for future advancements in automated programming and the completion of complex tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。